[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:28:33 dist-test-slave-njxd krb5kdc[1918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:28:33 dist-test-slave-njxd krb5kdc[1918](info): set up 2 sockets
Aug 10 22:28:33 dist-test-slave-njxd krb5kdc[1918](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 22:28:34.313418 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:28:34 dist-test-slave-njxd krb5kdc[1918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:28:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:28:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:28:34.445272 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:37879
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:37879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:34.570897 1932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:34.571234 1932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:34.571321 1932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:34.575464 1932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:28:34.575557 1932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:34.575598 1932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:34.575630 1932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:28:34.575663 1932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:28:34.581192 1932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:37879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:37879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:34.582775 1932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:34.584172 1932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:34.597821 1938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:34.597860 1937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:34.598006 1940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:34.598572 1932 server_base.cc:1034] running on GCE node
I20260810 22:28:34.599294 1932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:34.600848 1932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:34.602031 1932 hybrid_clock.cc:648] HybridClock initialized: now 1786400914602006 us; error 39 us; skew 500 ppm
Aug 10 22:28:34 dist-test-slave-njxd krb5kdc[1918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400914, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:34.610800 1932 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:28:34.612044 1932 webserver.cc:460] Webserver started at http://127.1.221.190:45913/ using document root <none> and password file <none>
I20260810 22:28:34.612998 1932 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:34.613097 1932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:34.613819 1932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:34.616756 1932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "f37d550410f744ae9c244e24014dad81"
format_stamp: "Formatted at 2026-08-10 22:28:34 on dist-test-slave-njxd"
server_key: "d42747d9736fce1be454a54f867f6a7d"
server_key_iv: "e342f4d3a710d357029ea42055942297"
server_key_version: "encryptionkey@0"
I20260810 22:28:34.617728 1932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "f37d550410f744ae9c244e24014dad81"
format_stamp: "Formatted at 2026-08-10 22:28:34 on dist-test-slave-njxd"
server_key: "d42747d9736fce1be454a54f867f6a7d"
server_key_iv: "e342f4d3a710d357029ea42055942297"
server_key_version: "encryptionkey@0"
I20260810 22:28:34.623106 1932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 22:28:34.626969 1948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:34.628674 1932 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:28:34.628885 1932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "f37d550410f744ae9c244e24014dad81"
format_stamp: "Formatted at 2026-08-10 22:28:34 on dist-test-slave-njxd"
server_key: "d42747d9736fce1be454a54f867f6a7d"
server_key_iv: "e342f4d3a710d357029ea42055942297"
server_key_version: "encryptionkey@0"
I20260810 22:28:34.629010 1932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:34.646400 1932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:34.650400 1932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:34.650686 1932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:34.666548 1932 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:37879
I20260810 22:28:34.666769 2008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:37879 every 8 connection(s)
I20260810 22:28:34.668051 1932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:28:34.673362 2009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:34.673516 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 1932
I20260810 22:28:34.673725 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:28:34.676162 1910 external_mini_cluster.cc:1428] Setting key fe0d6df35945e431ce7e8f65ac554057
I20260810 22:28:34.685566 2009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81: Bootstrap starting.
I20260810 22:28:34.690543 2009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:34.692239 2009 log.cc:824] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:34.695497 2009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81: No bootstrap required, opened a new log
Aug 10 22:28:34 dist-test-slave-njxd krb5kdc[1918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:34.702190 2009 raft_consensus.cc:348] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37d550410f744ae9c244e24014dad81" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37879 } }
I20260810 22:28:34.702679 2009 raft_consensus.cc:374] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:34.702812 2009 raft_consensus.cc:729] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37d550410f744ae9c244e24014dad81, State: Initialized, Role: FOLLOWER
I20260810 22:28:34.703732 2009 consensus_queue.cc:260] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37d550410f744ae9c244e24014dad81" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37879 } }
I20260810 22:28:34.703997 2009 raft_consensus.cc:388] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:28:34.704159 2009 raft_consensus.cc:482] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:28:34.704360 2009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:34.705814 2009 raft_consensus.cc:504] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37d550410f744ae9c244e24014dad81" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37879 } }
I20260810 22:28:34.706436 2009 leader_election.cc:302] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f37d550410f744ae9c244e24014dad81; no voters:
I20260810 22:28:34.706940 2009 leader_election.cc:288] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:28:34.707198 2014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:34.707653 2014 raft_consensus.cc:686] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [term 1 LEADER]: Becoming Leader. State: Replica: f37d550410f744ae9c244e24014dad81, State: Running, Role: LEADER
I20260810 22:28:34.708113 2014 consensus_queue.cc:237] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37d550410f744ae9c244e24014dad81" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37879 } }
I20260810 22:28:34.708925 2009 sys_catalog.cc:564] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:28:34.710135 2014 sys_catalog.cc:455] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f37d550410f744ae9c244e24014dad81. Latest consensus state: current_term: 1 leader_uuid: "f37d550410f744ae9c244e24014dad81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37d550410f744ae9c244e24014dad81" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37879 } } }
I20260810 22:28:34.710246 2014 sys_catalog.cc:458] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:34.710536 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:34.682251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.1.221.190:37879)
0810 22:28:34.683177 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:34.683247 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:34.684113 (+ 866us) server_negotiation.cc:407] Connection header received
0810 22:28:34.689398 (+ 5285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:34.689422 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:34.689990 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:34.690410 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:34.693185 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:34.695147 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:34.696559 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:34.697480 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:34.697702 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:34.702330 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:34.702366 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:34.702384 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:34.702429 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:34.705232 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:34.705973 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:34.705980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:34.705999 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:34.706184 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:34.706549 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:34.706553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:34.706556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:34.707255 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:34.707505 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:34.708600 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0810 22:28:34.708975 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":227,"threads_started":1}
I20260810 22:28:34.711017 2022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:28:34.714221 2016 sys_catalog.cc:455] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f37d550410f744ae9c244e24014dad81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f37d550410f744ae9c244e24014dad81" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37879 } } }
I20260810 22:28:34.714444 2016 sys_catalog.cc:458] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81 [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:34.719765 2022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:28:34.729399 2022 catalog_manager.cc:1347] Generated new cluster ID: 856891cebc5d43ec909aa45ab123c565
I20260810 22:28:34.729477 2022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:28:34.748972 2022 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:28:34.750087 2022 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:28:34.756383 2022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f37d550410f744ae9c244e24014dad81: Generated new TSK 0
I20260810 22:28:34.757258 2022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:28:34.800973 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37879
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:34.988193 2037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:34.988478 2037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:34.988528 2037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:34.992671 2037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:34.992733 2037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:34.992828 2037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:28:34.998518 2037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37879
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:34.999732 2037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:35.000818 2037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:35.013036 2042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:35.012986 2043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:35.012985 2045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:35.013443 2037 server_base.cc:1034] running on GCE node
I20260810 22:28:35.013970 2037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:35.014565 2037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:35.015739 2037 hybrid_clock.cc:648] HybridClock initialized: now 1786400915015707 us; error 30 us; skew 500 ppm
Aug 10 22:28:35 dist-test-slave-njxd krb5kdc[1918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400915, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:35.019183 2037 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:28:35.020064 2037 webserver.cc:460] Webserver started at http://127.1.221.129:41339/ using document root <none> and password file <none>
I20260810 22:28:35.020640 2037 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:35.020702 2037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:35.020939 2037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:35.023295 2037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "07362f38e8214b6585fce4ee4b655328"
format_stamp: "Formatted at 2026-08-10 22:28:35 on dist-test-slave-njxd"
server_key: "b0f2d6b8a22b7aed21fbc969fcbb7381"
server_key_iv: "84d1054d2ff00cda904025b040db94c6"
server_key_version: "encryptionkey@0"
I20260810 22:28:35.023957 2037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "07362f38e8214b6585fce4ee4b655328"
format_stamp: "Formatted at 2026-08-10 22:28:35 on dist-test-slave-njxd"
server_key: "b0f2d6b8a22b7aed21fbc969fcbb7381"
server_key_iv: "84d1054d2ff00cda904025b040db94c6"
server_key_version: "encryptionkey@0"
I20260810 22:28:35.028023 2037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 22:28:35.030617 2052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:35.031927 2037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 22:28:35.032033 2037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "07362f38e8214b6585fce4ee4b655328"
format_stamp: "Formatted at 2026-08-10 22:28:35 on dist-test-slave-njxd"
server_key: "b0f2d6b8a22b7aed21fbc969fcbb7381"
server_key_iv: "84d1054d2ff00cda904025b040db94c6"
server_key_version: "encryptionkey@0"
I20260810 22:28:35.032119 2037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:35.044631 2037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:35.047807 2037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:35.048002 2037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:35.048800 2037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:35.050009 2037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:35.050063 2037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:35.050104 2037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:35.050125 2037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:35.069550 2037 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:45299
I20260810 22:28:35.069692 2179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:45299 every 8 connection(s)
I20260810 22:28:35.070755 2037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:28:35.071012 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 2037
I20260810 22:28:35.071134 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:28:35.071516 1910 external_mini_cluster.cc:1428] Setting key 9ad8fc92880150c70bd1e343d69159ab
Aug 10 22:28:35 dist-test-slave-njxd krb5kdc[1918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400915, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260810 22:28:35.092676 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:35.074632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:53919 (local address 127.1.221.190:37879)
0810 22:28:35.074852 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:35.074857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:35.076205 (+ 1348us) server_negotiation.cc:407] Connection header received
0810 22:28:35.079539 (+ 3334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:35.079554 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:35.079626 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:35.079685 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:35.081520 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.082118 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.084516 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.084757 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.084849 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:35.088802 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:35.088805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:35.088807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:35.088835 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:35.090289 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:35.090925 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:35.090930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:35.090932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:35.091027 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:35.091416 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:35.091420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:35.091422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:35.091679 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:35.091791 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:35.092332 (+ 541us) server_negotiation.cc:299] Negotiation successful
0810 22:28:35.092489 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 22:28:35.093900 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:35.075220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37879 (local address 127.1.221.129:53919)
0810 22:28:35.076030 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:35.076100 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:35.079206 (+ 3106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:35.080209 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:35.080219 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:35.080716 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:35.081356 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:35.081373 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.082259 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.082264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:35.084337 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:35.084361 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.084995 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.085000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:35.085370 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:35.086175 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:35.086201 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:35.088537 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:35.090436 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:35.090442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:35.090457 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:35.090783 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:35.091147 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:35.091150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:35.091152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:35.091308 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:35.091791 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:35.091799 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:35.092083 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 22:28:35.092287 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 22:28:35.092651 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":217,"threads_started":1}
I20260810 22:28:35.095350 2180 heartbeater.cc:347] Connected to a master server at 127.1.221.190:37879
I20260810 22:28:35.095610 2180 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:35.096127 2180 heartbeater.cc:511] Master 127.1.221.190:37879 requested a full tablet report, sending...
I20260810 22:28:35.098524 1966 ts_manager.cc:194] Registered new tserver with Master: 07362f38e8214b6585fce4ee4b655328 (127.1.221.129:45299)
I20260810 22:28:35.100890 1966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:53919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:28:35.112411 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37879
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:35.242751 2185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:35.243062 2185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:35.243110 2185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:35.247246 2185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:35.247325 2185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:35.247390 2185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:28:35.253155 2185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37879
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:35.254380 2185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:35.255432 2185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:35.265830 2191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:35.265905 2190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:35.265837 2193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:35.266191 2185 server_base.cc:1034] running on GCE node
I20260810 22:28:35.266773 2185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:35.267441 2185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:35.268627 2185 hybrid_clock.cc:648] HybridClock initialized: now 1786400915268611 us; error 28 us; skew 500 ppm
Aug 10 22:28:35 dist-test-slave-njxd krb5kdc[1918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400915, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:35.272096 2185 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:28:35.273077 2185 webserver.cc:460] Webserver started at http://127.1.221.130:41797/ using document root <none> and password file <none>
I20260810 22:28:35.273772 2185 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:35.273867 2185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:35.274097 2185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:35.276569 2185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "897a57c7d13143b38903017cb34c2cca"
format_stamp: "Formatted at 2026-08-10 22:28:35 on dist-test-slave-njxd"
server_key: "23ae9a26364870ca676643abe124bb9e"
server_key_iv: "246629c5aeacd0d0ec07f1a274394eff"
server_key_version: "encryptionkey@0"
I20260810 22:28:35.277170 2185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "897a57c7d13143b38903017cb34c2cca"
format_stamp: "Formatted at 2026-08-10 22:28:35 on dist-test-slave-njxd"
server_key: "23ae9a26364870ca676643abe124bb9e"
server_key_iv: "246629c5aeacd0d0ec07f1a274394eff"
server_key_version: "encryptionkey@0"
I20260810 22:28:35.281235 2185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 22:28:35.283883 2202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:35.285145 2185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:28:35.285283 2185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "897a57c7d13143b38903017cb34c2cca"
format_stamp: "Formatted at 2026-08-10 22:28:35 on dist-test-slave-njxd"
server_key: "23ae9a26364870ca676643abe124bb9e"
server_key_iv: "246629c5aeacd0d0ec07f1a274394eff"
server_key_version: "encryptionkey@0"
I20260810 22:28:35.285395 2185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:35.307832 2185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:35.312055 2185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:35.312353 2185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:35.313139 2185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:35.314133 2185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:35.314203 2185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:35.314275 2185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:35.314321 2185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:35.333138 2185 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:40865
I20260810 22:28:35.333271 2326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:40865 every 8 connection(s)
I20260810 22:28:35.334292 2185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:28:35.341044 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 2185
I20260810 22:28:35.341141 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:28:35.341387 1910 external_mini_cluster.cc:1428] Setting key 0984b00c1c625ae04d4c6981cb0e91b4
Aug 10 22:28:35 dist-test-slave-njxd krb5kdc[1918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400915, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:35.350224 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:35.336599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:52291 (local address 127.1.221.190:37879)
0810 22:28:35.336792 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:35.336797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:35.337856 (+ 1059us) server_negotiation.cc:407] Connection header received
0810 22:28:35.339413 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:35.339415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:35.339465 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:35.339541 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:35.340797 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.341429 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.342936 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.343184 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.343288 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:35.346262 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:35.346284 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:35.346289 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:35.346329 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:35.348110 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:35.348626 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:35.348635 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:35.348636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:35.348717 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:35.349069 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:35.349072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:35.349074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:35.349261 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:35.349365 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:35.349925 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 22:28:35.350054 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 22:28:35.351100 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:35.337093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37879 (local address 127.1.221.130:52291)
0810 22:28:35.337708 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:35.337755 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:35.339213 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:35.339710 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:35.339717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:35.340147 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:35.340669 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:35.340685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.341561 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.341564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:35.342814 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:35.342825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.343346 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.343350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:35.343565 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:35.344284 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:35.344301 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:35.346074 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:35.348231 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:35.348236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:35.348249 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:35.348509 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:35.348814 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:35.348817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:35.348819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:35.348966 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:35.349367 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:35.349376 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:35.349680 (+ 304us) client_negotiation.cc:769] Sending connection context
0810 22:28:35.349866 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 22:28:35.350215 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":116,"threads_started":1}
I20260810 22:28:35.352435 2327 heartbeater.cc:347] Connected to a master server at 127.1.221.190:37879
I20260810 22:28:35.352777 2327 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:35.353513 2327 heartbeater.cc:511] Master 127.1.221.190:37879 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260810 22:28:35.354806 1966 ts_manager.cc:194] Registered new tserver with Master: 897a57c7d13143b38903017cb34c2cca (127.1.221.130:40865)
I20260810 22:28:35.355449 1966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:52291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:28:35.374856 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37879
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:35.522818 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:35.523118 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:35.523164 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:35.527336 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:35.527405 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:35.527469 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:28:35.533309 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37879
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:35.534580 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:35.535624 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:35.546911 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:35.546967 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:35.546980 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:35.547353 2334 server_base.cc:1034] running on GCE node
I20260810 22:28:35.547824 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:35.548506 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:35.549669 2334 hybrid_clock.cc:648] HybridClock initialized: now 1786400915549647 us; error 35 us; skew 500 ppm
Aug 10 22:28:35 dist-test-slave-njxd krb5kdc[1918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400915, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:35.553282 2334 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:28:35.554275 2334 webserver.cc:460] Webserver started at http://127.1.221.131:45965/ using document root <none> and password file <none>
I20260810 22:28:35.554896 2334 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:35.554991 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:35.555230 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:35.557543 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "8df5e562854445ef828228b595f27acb"
format_stamp: "Formatted at 2026-08-10 22:28:35 on dist-test-slave-njxd"
server_key: "5e30699265edda387d5cb98dd5f812eb"
server_key_iv: "fbad19585dde0c3a3a91bd184b10a58b"
server_key_version: "encryptionkey@0"
I20260810 22:28:35.558212 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "8df5e562854445ef828228b595f27acb"
format_stamp: "Formatted at 2026-08-10 22:28:35 on dist-test-slave-njxd"
server_key: "5e30699265edda387d5cb98dd5f812eb"
server_key_iv: "fbad19585dde0c3a3a91bd184b10a58b"
server_key_version: "encryptionkey@0"
I20260810 22:28:35.562232 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:28:35.564802 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:35.566205 2334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:28:35.566350 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "8df5e562854445ef828228b595f27acb"
format_stamp: "Formatted at 2026-08-10 22:28:35 on dist-test-slave-njxd"
server_key: "5e30699265edda387d5cb98dd5f812eb"
server_key_iv: "fbad19585dde0c3a3a91bd184b10a58b"
server_key_version: "encryptionkey@0"
I20260810 22:28:35.566471 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:35.590997 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:35.594933 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:35.595188 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:35.596050 2334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:35.597021 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:35.597092 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:35.597163 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:35.597209 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:35.613121 2334 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:43999
I20260810 22:28:35.613160 2478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:43999 every 8 connection(s)
I20260810 22:28:35.614223 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:28:35.623826 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 2334
I20260810 22:28:35.623968 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:28:35.624239 1910 external_mini_cluster.cc:1428] Setting key 741a43b84fc7f012577693a7ffd238c1
Aug 10 22:28:35 dist-test-slave-njxd krb5kdc[1918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400915, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:35.631304 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:35.616328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:53645 (local address 127.1.221.190:37879)
0810 22:28:35.616470 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:35.616474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:35.617840 (+ 1366us) server_negotiation.cc:407] Connection header received
0810 22:28:35.619627 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:35.619631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:35.619695 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:35.619809 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:35.621695 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.622241 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.623775 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.624104 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.624218 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:35.627804 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:35.627807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:35.627809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:35.627835 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:35.629080 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:35.629700 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:35.629703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:35.629704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:35.629786 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:35.630236 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:35.630239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:35.630240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:35.630459 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:35.630542 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:35.631037 (+ 495us) server_negotiation.cc:299] Negotiation successful
0810 22:28:35.631145 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:28:35.632246 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:35.616915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37879 (local address 127.1.221.131:53645)
0810 22:28:35.617658 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:35.617729 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:35.619366 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:35.620016 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:35.620024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:35.620582 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:35.621492 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:35.621510 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.622383 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.622386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:35.623633 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:35.623643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.624266 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.624270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:35.624548 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:35.625379 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:35.625403 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:35.627639 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:35.629213 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:35.629218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:35.629232 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:35.629530 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:35.629866 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:35.629869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:35.629870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:35.630015 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:35.630550 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:35.630555 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:35.630812 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 22:28:35.630968 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 22:28:35.631276 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":154,"threads_started":1}
I20260810 22:28:35.633525 2479 heartbeater.cc:347] Connected to a master server at 127.1.221.190:37879
I20260810 22:28:35.633845 2479 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:35.634389 2479 heartbeater.cc:511] Master 127.1.221.190:37879 requested a full tablet report, sending...
I20260810 22:28:35.635423 1966 ts_manager.cc:194] Registered new tserver with Master: 8df5e562854445ef828228b595f27acb (127.1.221.131:43999)
I20260810 22:28:35.636029 1966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:53645
I20260810 22:28:35.639243 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 22:28:35 dist-test-slave-njxd krb5kdc[1918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400915, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:28:35 dist-test-slave-njxd krb5kdc[1918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400915, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260810 22:28:35.675707 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:35.664305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.1.221.129:45299)
0810 22:28:35.664652 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:35.664659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:35.664688 (+ 29us) server_negotiation.cc:407] Connection header received
0810 22:28:35.664746 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:35.664753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:35.664960 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:35.665110 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:35.665839 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.667106 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.667933 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.668550 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.668684 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:35.671542 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:35.671555 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:35.671582 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:35.671616 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:35.673780 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:35.674221 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:35.674228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:35.674230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:35.674334 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:35.674603 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:35.674607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:35.674609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:35.674943 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:35.675159 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:35.675308 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 22:28:35.675485 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
W20260810 22:28:35.676858 2085 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37602
Aug 10 22:28:35 dist-test-slave-njxd krb5kdc[1918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:28:35 dist-test-slave-njxd krb5kdc[1918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260810 22:28:35.705178 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:35.696623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.1.221.129:45299)
0810 22:28:35.696803 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:35.696807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:35.696849 (+ 42us) server_negotiation.cc:407] Connection header received
0810 22:28:35.696997 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:35.697000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:35.697044 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:35.697118 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:35.697859 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.698386 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.699209 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:35.699450 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:35.699525 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:35.702000 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:35.702004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:35.702006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:35.702032 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:35.703524 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:35.703947 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:35.703951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:35.703953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:35.704037 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:35.704321 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:35.704325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:35.704327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:35.704516 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:35.704652 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:35.704883 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 22:28:35.705012 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 22:28:35.706576 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 2037
I20260810 22:28:35.716513 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 2185
I20260810 22:28:35.728472 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 2334
I20260810 22:28:35.742455 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 1932
2026-08-10T22:28:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2555 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:28:35 dist-test-slave-njxd krb5kdc[2493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:28:35 dist-test-slave-njxd krb5kdc[2493](info): set up 2 sockets
Aug 10 22:28:35 dist-test-slave-njxd krb5kdc[2493](info): commencing operation
krb5kdc: starting...
W20260810 22:28:36.838192 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:28:36 dist-test-slave-njxd krb5kdc[2493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:28:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:28:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:28:36.964504 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34265
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:34265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:37.082020 2506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:37.082320 2506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:37.082397 2506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:37.086594 2506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:28:37.086683 2506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:37.086719 2506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:37.086758 2506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:28:37.086793 2506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:28:37.092308 2506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:34265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:37.093456 2506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:37.094612 2506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:37.103397 2513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:37.103405 2515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:37.103397 2512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:37.103499 2506 server_base.cc:1034] running on GCE node
I20260810 22:28:37.104230 2506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:37.105365 2506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:37.106523 2506 hybrid_clock.cc:648] HybridClock initialized: now 1786400917106499 us; error 38 us; skew 500 ppm
Aug 10 22:28:37 dist-test-slave-njxd krb5kdc[2493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400917, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:37.109339 2506 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:28:37.110328 2506 webserver.cc:460] Webserver started at http://127.1.221.190:41737/ using document root <none> and password file <none>
I20260810 22:28:37.111035 2506 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:37.111140 2506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:37.111367 2506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:37.113531 2506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "01c381a411f5406290d9312a9ec590cf"
format_stamp: "Formatted at 2026-08-10 22:28:37 on dist-test-slave-njxd"
server_key: "5447b8c8f7b5e2959cf8f79f5f21a532"
server_key_iv: "10abf2fb24e0ea497112747bd10b33b7"
server_key_version: "encryptionkey@0"
I20260810 22:28:37.114205 2506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "01c381a411f5406290d9312a9ec590cf"
format_stamp: "Formatted at 2026-08-10 22:28:37 on dist-test-slave-njxd"
server_key: "5447b8c8f7b5e2959cf8f79f5f21a532"
server_key_iv: "10abf2fb24e0ea497112747bd10b33b7"
server_key_version: "encryptionkey@0"
I20260810 22:28:37.118178 2506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 22:28:37.120738 2523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:37.122139 2506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:28:37.122274 2506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "01c381a411f5406290d9312a9ec590cf"
format_stamp: "Formatted at 2026-08-10 22:28:37 on dist-test-slave-njxd"
server_key: "5447b8c8f7b5e2959cf8f79f5f21a532"
server_key_iv: "10abf2fb24e0ea497112747bd10b33b7"
server_key_version: "encryptionkey@0"
I20260810 22:28:37.122388 2506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:37.141292 2506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:37.144275 2506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:37.144488 2506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:37.151782 2506 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:34265
I20260810 22:28:37.151815 2580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:34265 every 8 connection(s)
I20260810 22:28:37.152755 2506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:28:37.155550 2581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:37.161051 2581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf: Bootstrap starting.
I20260810 22:28:37.161808 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 2506
I20260810 22:28:37.161918 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:28:37.162187 1910 external_mini_cluster.cc:1428] Setting key 7e6d92e2dd9fc8bfb6d2ddb5750b8f18
I20260810 22:28:37.164057 2581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:37.164831 2581 log.cc:824] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:37.167059 2581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf: No bootstrap required, opened a new log
Aug 10 22:28:37 dist-test-slave-njxd krb5kdc[2493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:37.171880 2581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c381a411f5406290d9312a9ec590cf" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34265 } }
I20260810 22:28:37.172103 2581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:37.172173 2581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c381a411f5406290d9312a9ec590cf, State: Initialized, Role: FOLLOWER
I20260810 22:28:37.172703 2581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c381a411f5406290d9312a9ec590cf" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34265 } }
I20260810 22:28:37.172886 2581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:28:37.172955 2581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:28:37.173089 2581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:37.174369 2581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c381a411f5406290d9312a9ec590cf" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34265 } }
I20260810 22:28:37.174785 2581 leader_election.cc:302] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01c381a411f5406290d9312a9ec590cf; no voters:
I20260810 22:28:37.175179 2581 leader_election.cc:288] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:28:37.175252 2586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:37.175443 2586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [term 1 LEADER]: Becoming Leader. State: Replica: 01c381a411f5406290d9312a9ec590cf, State: Running, Role: LEADER
I20260810 22:28:37.175738 2586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c381a411f5406290d9312a9ec590cf" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34265 } }
I20260810 22:28:37.176512 2581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:28:37.177887 2588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01c381a411f5406290d9312a9ec590cf. Latest consensus state: current_term: 1 leader_uuid: "01c381a411f5406290d9312a9ec590cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c381a411f5406290d9312a9ec590cf" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34265 } } }
I20260810 22:28:37.178022 2588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:37.178321 2587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01c381a411f5406290d9312a9ec590cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c381a411f5406290d9312a9ec590cf" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34265 } } }
I20260810 22:28:37.178412 2587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:37.178443 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.163530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.1.221.190:34265)
0810 22:28:37.163988 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:37.164004 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:37.164039 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:28:37.164714 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:37.164732 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:37.165116 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:37.165505 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:37.166401 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.167471 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.168670 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.169148 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.169268 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.172362 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:37.172368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:37.172385 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:37.172419 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:37.174770 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:37.175374 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:37.175381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:37.175388 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:37.175571 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:37.175923 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:37.175926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:37.175928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:37.176428 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:37.176647 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:37.176938 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 22:28:37.177280 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":131,"threads_started":1}
I20260810 22:28:37.178706 2598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:28:37.182082 2598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:28:37.187933 2598 catalog_manager.cc:1347] Generated new cluster ID: 3b4e7b79002d40d29395bbfc95745571
I20260810 22:28:37.188015 2598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:28:37.213841 2598 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:28:37.215090 2598 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:28:37.221959 2598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01c381a411f5406290d9312a9ec590cf: Generated new TSK 0
I20260810 22:28:37.222685 2598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:28:37.266865 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34265
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:37.384282 2610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:37.384579 2610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:37.384670 2610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:37.388767 2610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:37.388852 2610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:37.388947 2610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:28:37.394299 2610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34265
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:37.395429 2610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:37.396509 2610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:28:37.406270 2610 server_base.cc:1034] running on GCE node
W20260810 22:28:37.406201 2615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:37.406201 2616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:37.406208 2618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:37.406945 2610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:37.407578 2610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:37.408736 2610 hybrid_clock.cc:648] HybridClock initialized: now 1786400917408721 us; error 26 us; skew 500 ppm
Aug 10 22:28:37 dist-test-slave-njxd krb5kdc[2493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400917, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:37.411643 2610 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:28:37.412547 2610 webserver.cc:460] Webserver started at http://127.1.221.129:44083/ using document root <none> and password file <none>
I20260810 22:28:37.413115 2610 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:37.413206 2610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:37.413426 2610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:37.415544 2610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "7aec38749d6646d886f1b9f3cc221489"
format_stamp: "Formatted at 2026-08-10 22:28:37 on dist-test-slave-njxd"
server_key: "35cddd83e6aa0fb9ce5d0092fd00cf94"
server_key_iv: "7b4a7eb0dcb7fd36061109bff89e03a3"
server_key_version: "encryptionkey@0"
I20260810 22:28:37.416109 2610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "7aec38749d6646d886f1b9f3cc221489"
format_stamp: "Formatted at 2026-08-10 22:28:37 on dist-test-slave-njxd"
server_key: "35cddd83e6aa0fb9ce5d0092fd00cf94"
server_key_iv: "7b4a7eb0dcb7fd36061109bff89e03a3"
server_key_version: "encryptionkey@0"
I20260810 22:28:37.419929 2610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:28:37.422415 2626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:37.423748 2610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:28:37.423892 2610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "7aec38749d6646d886f1b9f3cc221489"
format_stamp: "Formatted at 2026-08-10 22:28:37 on dist-test-slave-njxd"
server_key: "35cddd83e6aa0fb9ce5d0092fd00cf94"
server_key_iv: "7b4a7eb0dcb7fd36061109bff89e03a3"
server_key_version: "encryptionkey@0"
I20260810 22:28:37.424010 2610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:37.435827 2610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:37.439040 2610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:37.439260 2610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:37.439916 2610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:37.440774 2610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:37.440845 2610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:37.440913 2610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:37.440960 2610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:37.450189 2610 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:40991
I20260810 22:28:37.450222 2751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:40991 every 8 connection(s)
I20260810 22:28:37.451128 2610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:28:37.454571 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 2610
I20260810 22:28:37.454694 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:28:37.454963 1910 external_mini_cluster.cc:1428] Setting key 1fe7f7a9cc802593e4772ab8d72ae5be
Aug 10 22:28:37 dist-test-slave-njxd krb5kdc[2493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400917, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:37.466210 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.452958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:43469 (local address 127.1.221.190:34265)
0810 22:28:37.453096 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:37.453100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:37.453990 (+ 890us) server_negotiation.cc:407] Connection header received
0810 22:28:37.455550 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:37.455554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:37.455611 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:37.455720 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:37.457152 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.458022 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.459212 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.459617 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.459743 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.462385 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:37.462388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:37.462390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:37.462414 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:37.463888 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:37.464518 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:37.464521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:37.464522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:37.464601 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:37.464997 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:37.464999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:37.465000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:37.465171 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:37.465248 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:37.465972 (+ 724us) server_negotiation.cc:299] Negotiation successful
0810 22:28:37.466085 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260810 22:28:37.467203 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.453307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34265 (local address 127.1.221.129:43469)
0810 22:28:37.453830 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:37.453876 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:37.455275 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:37.455918 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:37.455927 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:37.456478 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:37.456999 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:37.457014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.458147 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.458150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:37.459087 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:37.459097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.459765 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.459767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:37.459916 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.460523 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:37.460540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:37.462213 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:37.464025 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:37.464033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:37.464051 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:37.464418 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:37.464698 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:37.464704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:37.464707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:37.464900 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:37.465286 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:37.465295 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:37.465690 (+ 395us) client_negotiation.cc:769] Sending connection context
0810 22:28:37.465940 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 22:28:37.466302 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":123,"threads_started":1}
I20260810 22:28:37.468526 2752 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34265
I20260810 22:28:37.468840 2752 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260810 22:28:37.469420 2752 heartbeater.cc:511] Master 127.1.221.190:34265 requested a full tablet report, sending...
I20260810 22:28:37.471076 2541 ts_manager.cc:194] Registered new tserver with Master: 7aec38749d6646d886f1b9f3cc221489 (127.1.221.129:40991)
I20260810 22:28:37.472226 2541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:43469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:28:37.485319 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34265
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:37.601464 2758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:37.601832 2758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:37.601882 2758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:37.605926 2758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:37.605991 2758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:37.606096 2758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:28:37.611481 2758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34265
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:37.612637 2758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:37.613752 2758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:37.625563 2768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:37.625751 2767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:37.625563 2770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:37.625880 2758 server_base.cc:1034] running on GCE node
I20260810 22:28:37.626427 2758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:37.627203 2758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:37.628405 2758 hybrid_clock.cc:648] HybridClock initialized: now 1786400917628367 us; error 55 us; skew 500 ppm
Aug 10 22:28:37 dist-test-slave-njxd krb5kdc[2493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400917, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:37.631358 2758 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:28:37.632319 2758 webserver.cc:460] Webserver started at http://127.1.221.130:39225/ using document root <none> and password file <none>
I20260810 22:28:37.632905 2758 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:37.632993 2758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:37.633237 2758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:37.635583 2758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "3d719c5be4fb4deb9294baead9909340"
format_stamp: "Formatted at 2026-08-10 22:28:37 on dist-test-slave-njxd"
server_key: "693eb949aa3d7f8c40d06ec057f1ee58"
server_key_iv: "356388e00c67beaf2548b353412c05bc"
server_key_version: "encryptionkey@0"
I20260810 22:28:37.636178 2758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "3d719c5be4fb4deb9294baead9909340"
format_stamp: "Formatted at 2026-08-10 22:28:37 on dist-test-slave-njxd"
server_key: "693eb949aa3d7f8c40d06ec057f1ee58"
server_key_iv: "356388e00c67beaf2548b353412c05bc"
server_key_version: "encryptionkey@0"
I20260810 22:28:37.640336 2758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:28:37.643163 2778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:37.644639 2758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:28:37.644799 2758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "3d719c5be4fb4deb9294baead9909340"
format_stamp: "Formatted at 2026-08-10 22:28:37 on dist-test-slave-njxd"
server_key: "693eb949aa3d7f8c40d06ec057f1ee58"
server_key_iv: "356388e00c67beaf2548b353412c05bc"
server_key_version: "encryptionkey@0"
I20260810 22:28:37.644919 2758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:37.661957 2758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:37.665479 2758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:37.665757 2758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:37.666400 2758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:37.667233 2758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:37.667280 2758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:37.667321 2758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:37.667343 2758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:37.676827 2758 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:36239
I20260810 22:28:37.676891 2906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:36239 every 8 connection(s)
I20260810 22:28:37.678005 2758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:28:37.682832 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 2758
I20260810 22:28:37.682945 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:28:37.683218 1910 external_mini_cluster.cc:1428] Setting key 43149363801755a66afa44ea7ddbc472
Aug 10 22:28:37 dist-test-slave-njxd krb5kdc[2493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400917, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:37.692472 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.680142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:54123 (local address 127.1.221.190:34265)
0810 22:28:37.680278 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:37.680281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:37.681141 (+ 860us) server_negotiation.cc:407] Connection header received
0810 22:28:37.682593 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:37.682598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:37.682665 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:37.682777 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:37.684231 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.684883 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.686112 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.686344 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.686454 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.688898 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:37.688902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:37.688905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:37.688927 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:37.690420 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:37.690945 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:37.690949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:37.690951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:37.691041 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:37.691386 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:37.691391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:37.691393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:37.691575 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:37.691679 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:37.692178 (+ 499us) server_negotiation.cc:299] Negotiation successful
0810 22:28:37.692309 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260810 22:28:37.693383 2910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.680519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34265 (local address 127.1.221.130:54123)
0810 22:28:37.680973 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:37.681018 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:37.682338 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:37.682961 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:37.682970 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:37.683429 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:37.684079 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:37.684094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.685018 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.685022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:37.685986 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:37.685995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.686498 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.686502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:37.686669 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.687297 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:37.687313 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:37.688730 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:37.690544 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:37.690550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:37.690566 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:37.690838 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:37.691133 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:37.691136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:37.691139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:37.691288 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:37.691678 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:37.691687 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:37.691940 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 22:28:37.692135 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 22:28:37.692484 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260810 22:28:37.694591 2907 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34265
I20260810 22:28:37.694868 2907 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:37.695456 2907 heartbeater.cc:511] Master 127.1.221.190:34265 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260810 22:28:37.696533 2541 ts_manager.cc:194] Registered new tserver with Master: 3d719c5be4fb4deb9294baead9909340 (127.1.221.130:36239)
I20260810 22:28:37.697399 2541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:54123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:28:37.711191 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34265
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:37.826715 2914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:37.827020 2914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:37.827075 2914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:37.831290 2914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:37.831360 2914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:37.831425 2914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:28:37.836836 2914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:34265
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:37.838052 2914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:37.839048 2914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:37.848696 2920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:37.848696 2919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:37.848711 2922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:37.849124 2914 server_base.cc:1034] running on GCE node
I20260810 22:28:37.849555 2914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:37.850230 2914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:37.851397 2914 hybrid_clock.cc:648] HybridClock initialized: now 1786400917851379 us; error 28 us; skew 500 ppm
Aug 10 22:28:37 dist-test-slave-njxd krb5kdc[2493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400917, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:37.854355 2914 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:28:37.855264 2914 webserver.cc:460] Webserver started at http://127.1.221.131:45403/ using document root <none> and password file <none>
I20260810 22:28:37.855857 2914 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:37.855921 2914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:37.856149 2914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:37.858355 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "3386831f5f4748fbbf010dddd82a07c7"
format_stamp: "Formatted at 2026-08-10 22:28:37 on dist-test-slave-njxd"
server_key: "164ba4657874b79d5719d299db6872b0"
server_key_iv: "3a39d4f8403551e7b10dd18bc21bf5e5"
server_key_version: "encryptionkey@0"
I20260810 22:28:37.858948 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "3386831f5f4748fbbf010dddd82a07c7"
format_stamp: "Formatted at 2026-08-10 22:28:37 on dist-test-slave-njxd"
server_key: "164ba4657874b79d5719d299db6872b0"
server_key_iv: "3a39d4f8403551e7b10dd18bc21bf5e5"
server_key_version: "encryptionkey@0"
I20260810 22:28:37.862723 2914 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 22:28:37.865249 2929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:37.866662 2914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:28:37.866791 2914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "3386831f5f4748fbbf010dddd82a07c7"
format_stamp: "Formatted at 2026-08-10 22:28:37 on dist-test-slave-njxd"
server_key: "164ba4657874b79d5719d299db6872b0"
server_key_iv: "3a39d4f8403551e7b10dd18bc21bf5e5"
server_key_version: "encryptionkey@0"
I20260810 22:28:37.866899 2914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:37.893070 2914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:37.896553 2914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:37.896771 2914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:37.897434 2914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:37.898355 2914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:37.898404 2914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:37.898447 2914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:37.898463 2914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:37.907433 2914 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:43457
I20260810 22:28:37.907455 3056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:43457 every 8 connection(s)
I20260810 22:28:37.908420 2914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:28:37.908587 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 2914
I20260810 22:28:37.908710 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:28:37.909083 1910 external_mini_cluster.cc:1428] Setting key 3c618e4f525e9db77d33f8b3f142589a
Aug 10 22:28:37 dist-test-slave-njxd krb5kdc[2493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400917, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:37.922544 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.910489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:53017 (local address 127.1.221.190:34265)
0810 22:28:37.910610 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:37.910614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:37.911668 (+ 1054us) server_negotiation.cc:407] Connection header received
0810 22:28:37.913083 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:37.913086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:37.913137 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:37.913211 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:37.914426 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.914965 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.916129 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.916351 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.916433 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.919001 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:37.919004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:37.919006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:37.919032 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:37.920441 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:37.920932 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:37.920935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:37.920936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:37.921008 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:37.921310 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:37.921312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:37.921313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:37.921478 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:37.921554 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:37.922252 (+ 698us) server_negotiation.cc:299] Negotiation successful
0810 22:28:37.922371 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 22:28:37.923564 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.911009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34265 (local address 127.1.221.131:53017)
0810 22:28:37.911510 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:37.911562 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:37.912896 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:37.913342 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:37.913348 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:37.913771 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:37.914282 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:37.914296 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.915088 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.915091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:37.916017 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:37.916026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.916480 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.916483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:37.916649 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.917274 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:37.917290 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:37.918846 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:37.920565 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:37.920570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:37.920583 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:37.920848 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:37.921084 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:37.921086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:37.921088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:37.921229 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:37.921606 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:37.921613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:37.921962 (+ 349us) client_negotiation.cc:769] Sending connection context
0810 22:28:37.922197 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 22:28:37.922516 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":143,"threads_started":1}
I20260810 22:28:37.924688 3057 heartbeater.cc:347] Connected to a master server at 127.1.221.190:34265
I20260810 22:28:37.924908 3057 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:37.925424 3057 heartbeater.cc:511] Master 127.1.221.190:34265 requested a full tablet report, sending...
I20260810 22:28:37.926499 2541 ts_manager.cc:194] Registered new tserver with Master: 3386831f5f4748fbbf010dddd82a07c7 (127.1.221.131:43457)
I20260810 22:28:37.927116 2541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:53017
I20260810 22:28:37.935355 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:28:37.947332 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.939784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.1.221.190:34265)
0810 22:28:37.939940 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:37.939945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:37.940126 (+ 181us) server_negotiation.cc:407] Connection header received
0810 22:28:37.940306 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:37.940309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:37.940352 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:37.940433 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:37.941165 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.941757 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.942577 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.942800 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.942909 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.944297 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:37.944300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:37.944303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:37.944334 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:37.945531 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:37.945993 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:37.945998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:37.946000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:37.946084 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:37.946356 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:37.946360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:37.946362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:37.946589 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:37.946727 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:37.946981 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 22:28:37.947127 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:28:37.953506 2541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:28:37.956929 2541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:28:37.972707 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.966528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:43457 (local address 127.0.0.1:44852)
0810 22:28:37.966875 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:37.966933 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:37.967065 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:37.968087 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:37.968094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:37.968118 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:37.968304 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:37.968313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.970031 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.970034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:37.971194 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:37.971204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.972302 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.972305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:37.972423 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.972449 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:28:37.972500 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 22:28:37.972570 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260810 22:28:37.972719 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.966788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34540 (local address 127.1.221.129:40991)
0810 22:28:37.967791 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:37.967808 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:37.967830 (+ 22us) server_negotiation.cc:407] Connection header received
0810 22:28:37.967958 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:37.967966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:37.968165 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:37.968325 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:37.968678 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.969551 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.971104 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.972002 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.972114 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.972328 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:37.972454 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 22:28:37.972553 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":105,"threads_started":1}
I20260810 22:28:37.973228 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.967275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36239 (local address 127.0.0.1:35060)
0810 22:28:37.968269 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:37.968285 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:37.968470 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:37.968978 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:37.968981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:37.969000 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:37.969093 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:37.969099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.970538 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.970542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:37.971548 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:37.971555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.972743 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.972750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:37.972845 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.972876 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:28:37.972957 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 22:28:37.973043 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"thread_start_us":111,"threads_started":1}
I20260810 22:28:37.973649 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.966683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.1.221.131:43457)
0810 22:28:37.967460 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:37.967475 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:37.967498 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:28:37.967608 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:37.967614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:37.967830 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:37.968458 (+ 628us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:37.968620 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.969830 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.971359 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.972143 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.973144 (+ 1001us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.973284 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:37.973356 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 22:28:37.973455 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":203,"threads_started":1}
I20260810 22:28:37.973784 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.967685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.1.221.130:36239)
0810 22:28:37.968134 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:37.968147 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:37.968498 (+ 351us) server_negotiation.cc:407] Connection header received
0810 22:28:37.968626 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:37.968631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:37.968830 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:37.968985 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:37.969219 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.970389 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.971665 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.972445 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.972538 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.972676 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:37.973504 (+ 828us) server_negotiation.cc:299] Negotiation successful
0810 22:28:37.973628 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":144,"threads_started":1}
I20260810 22:28:37.974081 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:37.966528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:40991 (local address 127.0.0.1:34540)
0810 22:28:37.967419 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:37.967432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:37.967516 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:37.968313 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:37.968317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:37.968350 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:37.968553 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:37.968564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.969706 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.969711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:37.970982 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:37.970992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:37.972142 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:37.972146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:37.972255 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:37.972299 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 22:28:37.972425 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 22:28:37.972566 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":501,"thread_start_us":133,"threads_started":1}
I20260810 22:28:37.976526 2984 tablet_service.cc:1467] Processing CreateTablet for tablet bebaf481c9b04d088305b13d2d9fbf20 (DEFAULT_TABLE table=test-table [id=6382e15f0b0c4f18bf08e8171356e2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:37.977970 2984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebaf481c9b04d088305b13d2d9fbf20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:37.978642 2833 tablet_service.cc:1467] Processing CreateTablet for tablet bebaf481c9b04d088305b13d2d9fbf20 (DEFAULT_TABLE table=test-table [id=6382e15f0b0c4f18bf08e8171356e2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:37.978896 2678 tablet_service.cc:1467] Processing CreateTablet for tablet bebaf481c9b04d088305b13d2d9fbf20 (DEFAULT_TABLE table=test-table [id=6382e15f0b0c4f18bf08e8171356e2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:37.979668 2833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebaf481c9b04d088305b13d2d9fbf20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:37.979807 2678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bebaf481c9b04d088305b13d2d9fbf20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:37.985695 3074 tablet_bootstrap.cc:492] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489: Bootstrap starting.
I20260810 22:28:37.986724 3077 tablet_bootstrap.cc:492] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340: Bootstrap starting.
I20260810 22:28:37.987427 3076 tablet_bootstrap.cc:492] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7: Bootstrap starting.
I20260810 22:28:37.988365 3074 tablet_bootstrap.cc:654] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:37.989327 3074 log.cc:824] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:37.989368 3077 tablet_bootstrap.cc:654] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:37.990258 3076 tablet_bootstrap.cc:654] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:37.990836 3077 log.cc:824] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:37.991585 3076 log.cc:824] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:37.992053 3074 tablet_bootstrap.cc:492] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489: No bootstrap required, opened a new log
I20260810 22:28:37.992259 3074 ts_tablet_manager.cc:1397] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 22:28:37.992929 3077 tablet_bootstrap.cc:492] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340: No bootstrap required, opened a new log
I20260810 22:28:37.993130 3077 ts_tablet_manager.cc:1397] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260810 22:28:37.993701 3076 tablet_bootstrap.cc:492] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7: No bootstrap required, opened a new log
I20260810 22:28:37.993943 3076 ts_tablet_manager.cc:1397] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260810 22:28:37.997512 3077 raft_consensus.cc:348] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3386831f5f4748fbbf010dddd82a07c7" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43457 } } peers { permanent_uuid: "7aec38749d6646d886f1b9f3cc221489" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40991 } } peers { permanent_uuid: "3d719c5be4fb4deb9294baead9909340" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36239 } }
I20260810 22:28:37.997840 3077 raft_consensus.cc:374] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:37.997900 3077 raft_consensus.cc:729] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d719c5be4fb4deb9294baead9909340, State: Initialized, Role: FOLLOWER
I20260810 22:28:37.998361 3077 consensus_queue.cc:260] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3386831f5f4748fbbf010dddd82a07c7" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43457 } } peers { permanent_uuid: "7aec38749d6646d886f1b9f3cc221489" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40991 } } peers { permanent_uuid: "3d719c5be4fb4deb9294baead9909340" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36239 } }
I20260810 22:28:37.998306 3074 raft_consensus.cc:348] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3386831f5f4748fbbf010dddd82a07c7" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43457 } } peers { permanent_uuid: "7aec38749d6646d886f1b9f3cc221489" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40991 } } peers { permanent_uuid: "3d719c5be4fb4deb9294baead9909340" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36239 } }
I20260810 22:28:37.998570 3074 raft_consensus.cc:374] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:37.998629 3074 raft_consensus.cc:729] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aec38749d6646d886f1b9f3cc221489, State: Initialized, Role: FOLLOWER
I20260810 22:28:37.998487 3076 raft_consensus.cc:348] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3386831f5f4748fbbf010dddd82a07c7" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43457 } } peers { permanent_uuid: "7aec38749d6646d886f1b9f3cc221489" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40991 } } peers { permanent_uuid: "3d719c5be4fb4deb9294baead9909340" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36239 } }
I20260810 22:28:37.998734 3076 raft_consensus.cc:374] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:37.998785 3076 raft_consensus.cc:729] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3386831f5f4748fbbf010dddd82a07c7, State: Initialized, Role: FOLLOWER
I20260810 22:28:37.999078 3074 consensus_queue.cc:260] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3386831f5f4748fbbf010dddd82a07c7" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43457 } } peers { permanent_uuid: "7aec38749d6646d886f1b9f3cc221489" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40991 } } peers { permanent_uuid: "3d719c5be4fb4deb9294baead9909340" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36239 } }
I20260810 22:28:37.999116 3076 consensus_queue.cc:260] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3386831f5f4748fbbf010dddd82a07c7" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43457 } } peers { permanent_uuid: "7aec38749d6646d886f1b9f3cc221489" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40991 } } peers { permanent_uuid: "3d719c5be4fb4deb9294baead9909340" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36239 } }
I20260810 22:28:37.999370 3077 ts_tablet_manager.cc:1428] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:28:37.999822 2907 heartbeater.cc:503] Master 127.1.221.190:34265 was elected leader, sending a full tablet report...
I20260810 22:28:38.000062 3074 ts_tablet_manager.cc:1428] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260810 22:28:38.000062 3076 ts_tablet_manager.cc:1428] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260810 22:28:38.000401 2752 heartbeater.cc:503] Master 127.1.221.190:34265 was elected leader, sending a full tablet report...
I20260810 22:28:38.000530 3057 heartbeater.cc:503] Master 127.1.221.190:34265 was elected leader, sending a full tablet report...
W20260810 22:28:38.159627 3058 tablet.cc:2369] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:28:38.179261 2908 tablet.cc:2369] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:28:38.202579 2753 tablet.cc:2369] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:28:38.270108 3083 raft_consensus.cc:482] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:28:38.270357 3083 raft_consensus.cc:504] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3386831f5f4748fbbf010dddd82a07c7" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43457 } } peers { permanent_uuid: "7aec38749d6646d886f1b9f3cc221489" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40991 } } peers { permanent_uuid: "3d719c5be4fb4deb9294baead9909340" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36239 } }
I20260810 22:28:38.271574 3083 leader_election.cc:288] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7aec38749d6646d886f1b9f3cc221489 (127.1.221.129:40991), 3d719c5be4fb4deb9294baead9909340 (127.1.221.130:36239)
I20260810 22:28:38.275650 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:38.271858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36239 (local address 127.1.221.131:57889)
0810 22:28:38.272034 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:38.272052 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:38.272179 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:38.272551 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:38.272554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:38.272576 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:38.272703 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:38.272713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.273541 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.273544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:38.274477 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:38.274487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.275179 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.275183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:38.275303 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:38.275330 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:28:38.275408 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 22:28:38.275480 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":19}
I20260810 22:28:38.275662 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:38.272104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:57889 (local address 127.1.221.130:36239)
0810 22:28:38.272242 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:38.272247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:38.272265 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:28:38.272354 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:38.272358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:38.272429 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:38.272513 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:38.272830 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.273428 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.274616 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.275061 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.275157 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:38.275201 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:38.275437 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 22:28:38.275528 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:28:38.275779 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:38.271990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:50989 (local address 127.1.221.129:40991)
0810 22:28:38.272138 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:38.272143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:38.272452 (+ 309us) server_negotiation.cc:407] Connection header received
0810 22:28:38.272602 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:38.272606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:38.272660 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:38.272749 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:38.273067 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.273644 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.274743 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.275209 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.275326 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:38.275382 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:38.275580 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 22:28:38.275654 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 22:28:38.275827 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:38.271856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:40991 (local address 127.1.221.131:50989)
0810 22:28:38.272339 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:38.272353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:38.272484 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:38.272814 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:38.272817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:38.272836 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:38.272949 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:38.272955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.273745 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.273747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:38.274618 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:38.274629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.275328 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.275331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:38.275440 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:38.275466 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:28:38.275542 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 22:28:38.275608 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":54,"thread_start_us":143,"threads_started":1}
I20260810 22:28:38.276569 2856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebaf481c9b04d088305b13d2d9fbf20" candidate_uuid: "3386831f5f4748fbbf010dddd82a07c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d719c5be4fb4deb9294baead9909340" is_pre_election: true
I20260810 22:28:38.276595 2701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebaf481c9b04d088305b13d2d9fbf20" candidate_uuid: "3386831f5f4748fbbf010dddd82a07c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aec38749d6646d886f1b9f3cc221489" is_pre_election: true
I20260810 22:28:38.277087 2856 raft_consensus.cc:2454] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3386831f5f4748fbbf010dddd82a07c7 in term 0.
I20260810 22:28:38.277734 2930 leader_election.cc:302] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3386831f5f4748fbbf010dddd82a07c7, 3d719c5be4fb4deb9294baead9909340; no voters:
I20260810 22:28:38.278056 3083 raft_consensus.cc:2781] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:28:38.278142 3083 raft_consensus.cc:482] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:28:38.278205 3083 raft_consensus.cc:3037] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:38.279241 2701 raft_consensus.cc:2454] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3386831f5f4748fbbf010dddd82a07c7 in term 0.
I20260810 22:28:38.279489 3083 raft_consensus.cc:504] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3386831f5f4748fbbf010dddd82a07c7" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43457 } } peers { permanent_uuid: "7aec38749d6646d886f1b9f3cc221489" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40991 } } peers { permanent_uuid: "3d719c5be4fb4deb9294baead9909340" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36239 } }
I20260810 22:28:38.279891 3083 leader_election.cc:288] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [CANDIDATE]: Term 1 election: Requested vote from peers 7aec38749d6646d886f1b9f3cc221489 (127.1.221.129:40991), 3d719c5be4fb4deb9294baead9909340 (127.1.221.130:36239)
I20260810 22:28:38.280344 2856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebaf481c9b04d088305b13d2d9fbf20" candidate_uuid: "3386831f5f4748fbbf010dddd82a07c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d719c5be4fb4deb9294baead9909340"
I20260810 22:28:38.280360 2701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bebaf481c9b04d088305b13d2d9fbf20" candidate_uuid: "3386831f5f4748fbbf010dddd82a07c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aec38749d6646d886f1b9f3cc221489"
I20260810 22:28:38.280486 2701 raft_consensus.cc:3037] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:38.280488 2856 raft_consensus.cc:3037] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:38.281883 2701 raft_consensus.cc:2454] T bebaf481c9b04d088305b13d2d9fbf20 P 7aec38749d6646d886f1b9f3cc221489 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3386831f5f4748fbbf010dddd82a07c7 in term 1.
I20260810 22:28:38.281883 2856 raft_consensus.cc:2454] T bebaf481c9b04d088305b13d2d9fbf20 P 3d719c5be4fb4deb9294baead9909340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3386831f5f4748fbbf010dddd82a07c7 in term 1.
I20260810 22:28:38.282212 2930 leader_election.cc:302] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3386831f5f4748fbbf010dddd82a07c7, 3d719c5be4fb4deb9294baead9909340; no voters:
I20260810 22:28:38.282461 3083 raft_consensus.cc:2781] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:38.282694 3083 raft_consensus.cc:686] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [term 1 LEADER]: Becoming Leader. State: Replica: 3386831f5f4748fbbf010dddd82a07c7, State: Running, Role: LEADER
I20260810 22:28:38.283041 3083 consensus_queue.cc:237] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3386831f5f4748fbbf010dddd82a07c7" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43457 } } peers { permanent_uuid: "7aec38749d6646d886f1b9f3cc221489" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40991 } } peers { permanent_uuid: "3d719c5be4fb4deb9294baead9909340" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36239 } }
I20260810 22:28:38.286703 2540 catalog_manager.cc:5495] T bebaf481c9b04d088305b13d2d9fbf20 P 3386831f5f4748fbbf010dddd82a07c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3386831f5f4748fbbf010dddd82a07c7 (127.1.221.131). New cstate: current_term: 1 leader_uuid: "3386831f5f4748fbbf010dddd82a07c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3386831f5f4748fbbf010dddd82a07c7" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7aec38749d6646d886f1b9f3cc221489" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d719c5be4fb4deb9294baead9909340" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36239 } health_report { overall_health: UNKNOWN } } }
Aug 10 22:28:38 dist-test-slave-njxd krb5kdc[2493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260810 22:28:38.338141 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:38.328409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.1.221.129:40991)
0810 22:28:38.328586 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:38.328591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:38.328803 (+ 212us) server_negotiation.cc:407] Connection header received
0810 22:28:38.328963 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:38.328966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:38.329023 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:38.329112 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:38.329907 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.330443 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.331321 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.331557 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.331673 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:38.334388 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:38.334405 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:38.334422 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:38.334455 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:38.336161 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:38.336627 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:38.336634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:38.336635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:38.336721 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:38.337064 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:38.337068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:38.337069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:38.337360 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:38.337614 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:38.337829 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 22:28:38.337967 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 22:28:38 dist-test-slave-njxd krb5kdc[2493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:28:38 dist-test-slave-njxd krb5kdc[2493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260810 22:28:38.369484 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:38.360626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.1.221.129:40991)
0810 22:28:38.360814 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:38.360820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:38.360986 (+ 166us) server_negotiation.cc:407] Connection header received
0810 22:28:38.361172 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:38.361181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:38.361228 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:38.361321 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:38.362163 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.362713 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.363572 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.363821 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.363933 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:38.366471 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:38.366474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:38.366477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:38.366505 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:38.367808 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:38.368221 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:38.368224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:38.368225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:38.368311 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:38.368592 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:38.368594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:38.368595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:38.368762 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:38.368864 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:38.369127 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 22:28:38.369240 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Aug 10 22:28:38 dist-test-slave-njxd krb5kdc[2493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:28:38 dist-test-slave-njxd krb5kdc[2493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260810 22:28:38.398422 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:38.390140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.1.221.129:40991)
0810 22:28:38.390344 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:38.390348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:38.390417 (+ 69us) server_negotiation.cc:407] Connection header received
0810 22:28:38.390543 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:38.390546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:38.390594 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:38.390668 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:38.391335 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.391833 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.392734 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:38.392944 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:38.393021 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:38.395551 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:38.395555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:38.395557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:38.395583 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:38.396833 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:38.397218 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:38.397221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:38.397222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:38.397306 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:38.397534 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:38.397536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:38.397537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:38.397756 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:38.397863 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:38.398112 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 22:28:38.398217 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260810 22:28:38.400113 2656 tablet_service.cc:2908] Rejecting scan request for tablet bebaf481c9b04d088305b13d2d9fbf20: Uninitialized: clean time has not yet been initialized
I20260810 22:28:38.402554 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 2610
I20260810 22:28:38.412454 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 2758
I20260810 22:28:38.421976 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 2914
I20260810 22:28:38.431476 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 2506
2026-08-10T22:28:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2689 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:28:38 dist-test-slave-njxd krb5kdc[3102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:28:38 dist-test-slave-njxd krb5kdc[3102](info): set up 2 sockets
Aug 10 22:28:38 dist-test-slave-njxd krb5kdc[3102](info): commencing operation
krb5kdc: starting...
W20260810 22:28:39.557260 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:28:39 dist-test-slave-njxd krb5kdc[3102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:28:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:28:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:28:39.694753 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:41619
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:41619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:39.809259 3115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:39.809621 3115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:39.809666 3115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:39.813719 3115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:28:39.813782 3115 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 22:28:39.813799 3115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:39.813817 3115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:39.813834 3115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:28:39.813849 3115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:28:39.819203 3115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:41619
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:41619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:39.820381 3115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:39.821442 3115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:39.830127 3121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:39.830127 3125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:39.830168 3122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:39.830605 3115 server_base.cc:1034] running on GCE node
I20260810 22:28:39.831209 3115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:39.832356 3115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:39.833544 3115 hybrid_clock.cc:648] HybridClock initialized: now 1786400919833528 us; error 28 us; skew 500 ppm
Aug 10 22:28:39 dist-test-slave-njxd krb5kdc[3102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400919, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:39.842034 3115 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:28:39.843076 3115 webserver.cc:460] Webserver started at http://127.1.221.190:38133/ using document root <none> and password file <none>
I20260810 22:28:39.843639 3115 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:39.843700 3115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:39.843891 3115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:39.846149 3115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "abe1122d7dae4acc9034e4e48838d5e1"
format_stamp: "Formatted at 2026-08-10 22:28:39 on dist-test-slave-njxd"
server_key: "3ad372b84cd2326feee84efd22fa02ad"
server_key_iv: "b1c078bb3a73290719272faa5192277d"
server_key_version: "encryptionkey@0"
I20260810 22:28:39.846704 3115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "abe1122d7dae4acc9034e4e48838d5e1"
format_stamp: "Formatted at 2026-08-10 22:28:39 on dist-test-slave-njxd"
server_key: "3ad372b84cd2326feee84efd22fa02ad"
server_key_iv: "b1c078bb3a73290719272faa5192277d"
server_key_version: "encryptionkey@0"
I20260810 22:28:39.850665 3115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:28:39.853170 3132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:39.854578 3115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:28:39.854722 3115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "abe1122d7dae4acc9034e4e48838d5e1"
format_stamp: "Formatted at 2026-08-10 22:28:39 on dist-test-slave-njxd"
server_key: "3ad372b84cd2326feee84efd22fa02ad"
server_key_iv: "b1c078bb3a73290719272faa5192277d"
server_key_version: "encryptionkey@0"
I20260810 22:28:39.854979 3115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:39.876852 3115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:39.880053 3115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:39.880287 3115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:39.888967 3115 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:41619
I20260810 22:28:39.888988 3203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:41619 every 8 connection(s)
I20260810 22:28:39.890041 3115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:28:39.891523 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 3115
I20260810 22:28:39.891652 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:28:39.891945 1910 external_mini_cluster.cc:1428] Setting key 10f9589266f81845c4c264d708d02887
I20260810 22:28:39.893321 3204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:39.900686 3204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1: Bootstrap starting.
Aug 10 22:28:39 dist-test-slave-njxd krb5kdc[3102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:39.903450 3204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:39.905078 3204 log.cc:824] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:39.907461 3204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1: No bootstrap required, opened a new log
I20260810 22:28:39.907631 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:39.893475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.1.221.190:41619)
0810 22:28:39.893937 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:39.893962 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:39.894007 (+ 45us) server_negotiation.cc:407] Connection header received
0810 22:28:39.894883 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:39.894909 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:39.895344 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:39.895732 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:39.896409 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:39.897954 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:39.898852 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:39.899580 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:39.899762 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:39.902264 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:39.902270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:39.902285 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:39.902318 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:39.904488 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:39.904941 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:39.904949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:39.904958 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:39.905100 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:39.905406 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:39.905409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:39.905411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:39.905902 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:39.906135 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:39.906341 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 22:28:39.906672 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":130,"threads_started":1}
I20260810 22:28:39.910916 3204 raft_consensus.cc:348] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe1122d7dae4acc9034e4e48838d5e1" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41619 } }
I20260810 22:28:39.911137 3204 raft_consensus.cc:374] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:39.911173 3204 raft_consensus.cc:729] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe1122d7dae4acc9034e4e48838d5e1, State: Initialized, Role: FOLLOWER
I20260810 22:28:39.911553 3204 consensus_queue.cc:260] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe1122d7dae4acc9034e4e48838d5e1" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41619 } }
I20260810 22:28:39.911670 3204 raft_consensus.cc:388] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:28:39.911715 3204 raft_consensus.cc:482] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:28:39.911782 3204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:39.912653 3204 raft_consensus.cc:504] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe1122d7dae4acc9034e4e48838d5e1" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41619 } }
I20260810 22:28:39.912940 3204 leader_election.cc:302] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abe1122d7dae4acc9034e4e48838d5e1; no voters:
I20260810 22:28:39.913194 3204 leader_election.cc:288] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:28:39.913331 3209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:39.913569 3209 raft_consensus.cc:686] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [term 1 LEADER]: Becoming Leader. State: Replica: abe1122d7dae4acc9034e4e48838d5e1, State: Running, Role: LEADER
I20260810 22:28:39.913920 3209 consensus_queue.cc:237] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe1122d7dae4acc9034e4e48838d5e1" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41619 } }
I20260810 22:28:39.914542 3204 sys_catalog.cc:564] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:28:39.915897 3211 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe1122d7dae4acc9034e4e48838d5e1. Latest consensus state: current_term: 1 leader_uuid: "abe1122d7dae4acc9034e4e48838d5e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe1122d7dae4acc9034e4e48838d5e1" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41619 } } }
I20260810 22:28:39.916023 3211 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:39.916356 3210 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abe1122d7dae4acc9034e4e48838d5e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe1122d7dae4acc9034e4e48838d5e1" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41619 } } }
I20260810 22:28:39.916455 3210 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1 [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:39.916421 3220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:28:39.919626 3220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:28:39.925551 3220 catalog_manager.cc:1347] Generated new cluster ID: ddd4488c38dc4a54bb1a4a719aee86f9
I20260810 22:28:39.925654 3220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:28:39.936614 3220 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:28:39.937906 3220 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:28:39.943372 3220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1: Generated new TSK 0
I20260810 22:28:39.944209 3220 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:28:39.976368 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41619
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:40.095463 3231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:40.095746 3231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:40.095850 3231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:40.099838 3231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:40.099917 3231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:40.099997 3231 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:28:40.100073 3231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:28:40.105758 3231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:40.107065 3231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:40.108317 3231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:40.121086 3238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:40.121071 3237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:40.121071 3241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:40.121906 3231 server_base.cc:1034] running on GCE node
I20260810 22:28:40.122525 3231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:40.123526 3231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:40.124744 3231 hybrid_clock.cc:648] HybridClock initialized: now 1786400920124706 us; error 51 us; skew 500 ppm
Aug 10 22:28:40 dist-test-slave-njxd krb5kdc[3102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400920, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:40.127892 3231 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:28:40.128862 3231 webserver.cc:460] Webserver started at http://127.1.221.129:37167/ using document root <none> and password file <none>
I20260810 22:28:40.129470 3231 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:40.129537 3231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:40.129822 3231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:40.132043 3231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "10958515cd76484292c779df099a7f3a"
format_stamp: "Formatted at 2026-08-10 22:28:40 on dist-test-slave-njxd"
server_key: "ca19f3df5608ea2c4e8016b4ae2b9793"
server_key_iv: "e8d03aa38f48e34494cf8f4489b4dfcf"
server_key_version: "encryptionkey@0"
I20260810 22:28:40.132663 3231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "10958515cd76484292c779df099a7f3a"
format_stamp: "Formatted at 2026-08-10 22:28:40 on dist-test-slave-njxd"
server_key: "ca19f3df5608ea2c4e8016b4ae2b9793"
server_key_iv: "e8d03aa38f48e34494cf8f4489b4dfcf"
server_key_version: "encryptionkey@0"
I20260810 22:28:40.136839 3231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 22:28:40.139608 3248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:40.140901 3231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:28:40.141139 3231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "10958515cd76484292c779df099a7f3a"
format_stamp: "Formatted at 2026-08-10 22:28:40 on dist-test-slave-njxd"
server_key: "ca19f3df5608ea2c4e8016b4ae2b9793"
server_key_iv: "e8d03aa38f48e34494cf8f4489b4dfcf"
server_key_version: "encryptionkey@0"
I20260810 22:28:40.141261 3231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:40.152263 3231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:40.155212 3231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:40.155411 3231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:40.156855 3231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:40.156903 3231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:40.156965 3231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:40.156998 3231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:28:40 dist-test-slave-njxd krb5kdc[3102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400920, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:40.168352 3231 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:46543
I20260810 22:28:40.168512 3376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:46543 every 8 connection(s)
I20260810 22:28:40.169303 3231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:28:40.172209 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.157794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:37977 (local address 127.1.221.190:41619)
0810 22:28:40.157931 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:40.157935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:40.159049 (+ 1114us) server_negotiation.cc:407] Connection header received
0810 22:28:40.160211 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:40.160217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:40.160305 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:40.160373 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:40.162157 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.163066 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.164969 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.165282 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.165366 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.168635 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:40.168639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:40.168641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:40.168671 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:40.170386 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.170917 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.170919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.170921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.170998 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.171297 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.171299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.171300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.171471 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:40.171545 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:40.171985 (+ 440us) server_negotiation.cc:299] Negotiation successful
0810 22:28:40.172089 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 22:28:40.172971 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.158363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41619 (local address 127.1.221.129:37977)
0810 22:28:40.158859 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:40.158903 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:40.159923 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:40.160813 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:40.160825 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:40.161233 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:40.161975 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.161998 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.163224 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.163231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.164811 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.164824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.165468 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.165474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.165737 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.166525 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:40.166544 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:40.168468 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:40.170467 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:40.170471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:40.170480 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:40.170818 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:40.171074 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:40.171076 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:40.171077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:40.171206 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:40.171553 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:40.171558 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:40.171783 (+ 225us) client_negotiation.cc:769] Sending connection context
0810 22:28:40.171927 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 22:28:40.172165 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":146,"threads_started":1}
I20260810 22:28:40.173606 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 3231
I20260810 22:28:40.173741 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:28:40.174041 1910 external_mini_cluster.cc:1428] Setting key e033d9f57c22c00664aa3c9e8401bdb9
I20260810 22:28:40.174314 3377 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41619
I20260810 22:28:40.174467 3377 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:40.175052 3377 heartbeater.cc:511] Master 127.1.221.190:41619 requested a full tablet report, sending...
I20260810 22:28:40.176673 3149 ts_manager.cc:194] Registered new tserver with Master: 10958515cd76484292c779df099a7f3a (127.1.221.129:46543)
I20260810 22:28:40.178293 3149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:37977
I20260810 22:28:40.184651 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.177709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:39233 (local address 127.1.221.190:41619)
0810 22:28:40.177848 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:40.177854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:40.177868 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:28:40.177909 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:40.177913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:40.177957 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:40.178041 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:40.178562 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.179360 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.180279 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.180507 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.180610 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.181731 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:40.181735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:40.181748 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:40.181774 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:40.183079 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.183518 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.183522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.183524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.183598 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.183861 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.183865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.183867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.184028 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:40.184125 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:40.184389 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 22:28:40.184497 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:28:40.184665 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.176489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41619 (local address 127.1.221.129:39233)
0810 22:28:40.176812 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:40.176828 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:40.176911 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:40.178080 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:40.178083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:40.178305 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:40.178443 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.178450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.179476 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.179479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.180141 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.180149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.180617 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.180620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.180712 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.181189 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:40.181206 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:40.181522 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:40.183225 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:40.183228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:40.183230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:40.183420 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:40.183686 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:40.183689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:40.183691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:40.183774 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:40.184119 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:40.184122 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:40.184223 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 22:28:40.184334 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 22:28:40.184473 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":62,"thread_start_us":146,"threads_started":1}
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:28:40.203943 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41619
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:40.321233 3386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:40.321514 3386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:40.321563 3386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:40.325749 3386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:40.325816 3386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:40.325860 3386 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:28:40.325893 3386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:28:40.331471 3386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:40.332598 3386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:40.333712 3386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:40.344152 3394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:40.344197 3393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:40.344308 3386 server_base.cc:1034] running on GCE node
W20260810 22:28:40.344254 3396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:40.344861 3386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:40.345642 3386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:40.346827 3386 hybrid_clock.cc:648] HybridClock initialized: now 1786400920346792 us; error 47 us; skew 500 ppm
Aug 10 22:28:40 dist-test-slave-njxd krb5kdc[3102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400920, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:40.350567 3386 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:28:40.351758 3386 webserver.cc:460] Webserver started at http://127.1.221.130:40907/ using document root <none> and password file <none>
I20260810 22:28:40.352316 3386 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:40.352377 3386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:40.352550 3386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:40.354825 3386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "823ac8d5523f4526ba4bd50e7eeeb7be"
format_stamp: "Formatted at 2026-08-10 22:28:40 on dist-test-slave-njxd"
server_key: "e1b0316cd54a8d786e3c8d2a99bbff38"
server_key_iv: "329b816ebcd449ea87492057a8bbdd42"
server_key_version: "encryptionkey@0"
I20260810 22:28:40.355434 3386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "823ac8d5523f4526ba4bd50e7eeeb7be"
format_stamp: "Formatted at 2026-08-10 22:28:40 on dist-test-slave-njxd"
server_key: "e1b0316cd54a8d786e3c8d2a99bbff38"
server_key_iv: "329b816ebcd449ea87492057a8bbdd42"
server_key_version: "encryptionkey@0"
I20260810 22:28:40.359551 3386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 22:28:40.362165 3405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:40.363483 3386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:28:40.363672 3386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "823ac8d5523f4526ba4bd50e7eeeb7be"
format_stamp: "Formatted at 2026-08-10 22:28:40 on dist-test-slave-njxd"
server_key: "e1b0316cd54a8d786e3c8d2a99bbff38"
server_key_iv: "329b816ebcd449ea87492057a8bbdd42"
server_key_version: "encryptionkey@0"
I20260810 22:28:40.363801 3386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:40.386984 3386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:40.390946 3386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:40.391197 3386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:40.392930 3386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:40.393010 3386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:40.393092 3386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:40.393138 3386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:28:40 dist-test-slave-njxd krb5kdc[3102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400920, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:40.405651 3386 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:37831
I20260810 22:28:40.405699 3536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:37831 every 8 connection(s)
I20260810 22:28:40.407114 3386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:28:40.409334 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.393908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:38101 (local address 127.1.221.190:41619)
0810 22:28:40.394039 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:40.394043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:40.395288 (+ 1245us) server_negotiation.cc:407] Connection header received
0810 22:28:40.396316 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:40.396319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:40.396382 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:40.396469 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:40.398241 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.399075 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.401219 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.401617 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.401733 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.405033 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:40.405038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:40.405042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:40.405086 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:40.406975 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.407588 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.407595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.407597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.407705 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.408129 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.408134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.408136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.408391 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:40.408503 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:40.409010 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 22:28:40.409143 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:28:40.410207 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.394587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41619 (local address 127.1.221.130:38101)
0810 22:28:40.395130 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:40.395178 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:40.396099 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:40.396808 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:40.396818 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:40.397333 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:40.398072 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.398095 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.399223 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.399230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.401066 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.401085 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.401796 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.401805 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.402025 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.402694 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:40.402719 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:40.404835 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:40.407120 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:40.407125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:40.407144 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:40.407462 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:40.407831 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:40.407835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:40.407837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:40.408034 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:40.408502 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:40.408509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:40.408773 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 22:28:40.408965 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 22:28:40.409296 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":168,"threads_started":1}
I20260810 22:28:40.411404 3537 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41619
I20260810 22:28:40.411550 3537 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:40.411643 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 3386
I20260810 22:28:40.411773 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:28:40.412034 1910 external_mini_cluster.cc:1428] Setting key cb9a1b46ff60a7524416a700b391d512
I20260810 22:28:40.412128 3537 heartbeater.cc:511] Master 127.1.221.190:41619 requested a full tablet report, sending...
I20260810 22:28:40.413856 3150 ts_manager.cc:194] Registered new tserver with Master: 823ac8d5523f4526ba4bd50e7eeeb7be (127.1.221.130:37831)
I20260810 22:28:40.414692 3150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:38101
I20260810 22:28:40.422427 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.413751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:54701 (local address 127.1.221.190:41619)
0810 22:28:40.413857 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:40.413861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:40.414118 (+ 257us) server_negotiation.cc:407] Connection header received
0810 22:28:40.414226 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:40.414231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:40.414290 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:40.414378 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:40.414902 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.415775 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.417079 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.417466 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.417639 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.418995 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:40.418998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:40.419000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:40.419023 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:40.420471 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.421006 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.421008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.421009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.421083 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.421382 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.421385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.421386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.421678 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:40.421758 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:40.422181 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 22:28:40.422308 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 22:28:40.422452 3543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.413708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41619 (local address 127.1.221.130:54701)
0810 22:28:40.414020 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:40.414035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:40.414145 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:40.414413 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:40.414416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:40.414649 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:40.414779 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.414787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.415894 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.415898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.416906 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.416917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.417613 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.417621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.417697 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.418278 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:40.418299 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:40.418852 (+ 553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:40.420580 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:40.420586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:40.420591 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:40.420899 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:40.421172 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:40.421176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:40.421179 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:40.421287 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:40.421777 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:40.421780 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:40.421895 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 22:28:40.422026 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 22:28:40.422194 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:28:40.444635 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41619
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:40.561128 3546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:40.561407 3546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:40.561455 3546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:40.565474 3546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:40.565551 3546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:40.565653 3546 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:28:40.565701 3546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:28:40.571244 3546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:41619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:40.572358 3546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:40.573400 3546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:40.583546 3551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:40.583590 3552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:40.583559 3554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:40.583714 3546 server_base.cc:1034] running on GCE node
I20260810 22:28:40.584348 3546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:40.584986 3546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:40.586143 3546 hybrid_clock.cc:648] HybridClock initialized: now 1786400920586126 us; error 27 us; skew 500 ppm
Aug 10 22:28:40 dist-test-slave-njxd krb5kdc[3102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400920, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:40.588955 3546 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:28:40.589953 3546 webserver.cc:460] Webserver started at http://127.1.221.131:44051/ using document root <none> and password file <none>
I20260810 22:28:40.590533 3546 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:40.590596 3546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:40.590858 3546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:40.593072 3546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "a22c99ccdf344b229edd87a583455766"
format_stamp: "Formatted at 2026-08-10 22:28:40 on dist-test-slave-njxd"
server_key: "b2b9d63705ec8017db69e8ec58e208a0"
server_key_iv: "378e428e877d5f4f961fd39c9d021080"
server_key_version: "encryptionkey@0"
I20260810 22:28:40.593705 3546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "a22c99ccdf344b229edd87a583455766"
format_stamp: "Formatted at 2026-08-10 22:28:40 on dist-test-slave-njxd"
server_key: "b2b9d63705ec8017db69e8ec58e208a0"
server_key_iv: "378e428e877d5f4f961fd39c9d021080"
server_key_version: "encryptionkey@0"
I20260810 22:28:40.597558 3546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 22:28:40.600021 3561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:40.601259 3546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:28:40.601404 3546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "a22c99ccdf344b229edd87a583455766"
format_stamp: "Formatted at 2026-08-10 22:28:40 on dist-test-slave-njxd"
server_key: "b2b9d63705ec8017db69e8ec58e208a0"
server_key_iv: "378e428e877d5f4f961fd39c9d021080"
server_key_version: "encryptionkey@0"
I20260810 22:28:40.601507 3546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:40.615751 3546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:40.618961 3546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:40.619127 3546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:40.620636 3546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:40.620689 3546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:40.620738 3546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:40.620788 3546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:28:40 dist-test-slave-njxd krb5kdc[3102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400920, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:40.632123 3546 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:35433
I20260810 22:28:40.632154 3698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:35433 every 8 connection(s)
I20260810 22:28:40.633126 3546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:28:40.634760 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.621452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:41813 (local address 127.1.221.190:41619)
0810 22:28:40.621637 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:40.621644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:40.622576 (+ 932us) server_negotiation.cc:407] Connection header received
0810 22:28:40.623734 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:40.623738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:40.623811 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:40.623910 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:40.625670 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.626353 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.627599 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.627879 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.627995 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.630835 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:40.630841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:40.630848 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:40.630885 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:40.632541 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.633126 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.633130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.633132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.633250 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.633664 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.633668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.633669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.633923 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:40.634023 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:40.634527 (+ 504us) server_negotiation.cc:299] Negotiation successful
0810 22:28:40.634642 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:28:40.635579 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.621926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41619 (local address 127.1.221.131:41813)
0810 22:28:40.622403 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:40.622456 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:40.623470 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:40.624144 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:40.624158 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:40.624678 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:40.625460 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.625485 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.626485 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.626489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.627480 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.627494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.628074 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.628080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.628249 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.628844 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:40.628863 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:40.630675 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:40.632659 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:40.632667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:40.632684 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:40.633017 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:40.633352 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:40.633355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:40.633357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:40.633503 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:40.634038 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:40.634044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:40.634308 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 22:28:40.634458 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 22:28:40.634723 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":127,"threads_started":1}
I20260810 22:28:40.636802 3699 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41619
I20260810 22:28:40.636965 3699 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:40.637458 3699 heartbeater.cc:511] Master 127.1.221.190:41619 requested a full tablet report, sending...
I20260810 22:28:40.638574 3149 ts_manager.cc:194] Registered new tserver with Master: a22c99ccdf344b229edd87a583455766 (127.1.221.131:35433)
I20260810 22:28:40.639358 3149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:41813
I20260810 22:28:40.642130 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 3546
I20260810 22:28:40.642251 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:28:40.642525 1910 external_mini_cluster.cc:1428] Setting key 9893fc1d2fc6aa3df143c2c672c8228a
I20260810 22:28:40.644675 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:28:40.646512 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.639082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:58531 (local address 127.1.221.190:41619)
0810 22:28:40.639212 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:40.639217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:40.639483 (+ 266us) server_negotiation.cc:407] Connection header received
0810 22:28:40.639586 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:40.639588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:40.639627 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:40.639716 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:40.640280 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.640782 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.641619 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.641842 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.641950 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.643201 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:40.643204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:40.643207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:40.643237 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:40.644881 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.645294 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.645299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.645301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.645405 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.645756 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.645758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.645760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.645920 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:40.645996 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:40.646307 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 22:28:40.646399 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:28:40.646726 3705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.638980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41619 (local address 127.1.221.131:58531)
0810 22:28:40.639386 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:40.639402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:40.639501 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:40.639835 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:40.639839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:40.640046 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:40.640168 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.640174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.640906 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.640909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.641481 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.641488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.641955 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.641958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.642059 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.642632 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:40.642648 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:40.643076 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:40.644990 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:40.644993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:40.644995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:40.645194 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:40.645510 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:40.645516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:40.645521 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:40.645665 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:40.646012 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:40.646018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:40.646139 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 22:28:40.646258 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 22:28:40.646438 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
Aug 10 22:28:40 dist-test-slave-njxd krb5kdc[3102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400920, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:28:40 dist-test-slave-njxd krb5kdc[3102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400920, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:40.675668 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.666592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36590 (local address 127.1.221.190:41619)
0810 22:28:40.666754 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:40.666759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:40.666775 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:28:40.666829 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:40.666833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:40.666893 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:40.666989 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:40.668052 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.668689 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.669858 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.670263 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.670398 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.672814 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:40.672817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:40.672820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:40.672849 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:40.674198 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.674608 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.674610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.674611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.674684 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:40.674889 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:40.674891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:40.674892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:40.675047 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:40.675144 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:40.675388 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 22:28:40.675510 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:28:40.678694 3149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:28:40.681761 3149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:28:40.696893 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.690702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:35433 (local address 127.0.0.1:52592)
0810 22:28:40.691663 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:40.691717 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:40.691875 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:40.692655 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:40.692663 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:40.692679 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:40.692798 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.692804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.694071 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.694074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.695504 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.695516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.696480 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.696483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.696571 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.696602 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:28:40.696681 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 22:28:40.696766 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":84,"thread_start_us":85,"threads_started":1}
I20260810 22:28:40.696918 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.690981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.1.221.131:35433)
0810 22:28:40.691474 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:40.691490 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:40.691995 (+ 505us) server_negotiation.cc:407] Connection header received
0810 22:28:40.692105 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:40.692111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:40.692318 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:40.692487 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:40.692936 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.693920 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.695643 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.696371 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.696486 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.696658 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:40.696714 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 22:28:40.696802 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":117,"threads_started":1}
I20260810 22:28:40.697503 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.690702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:37831 (local address 127.0.0.1:33086)
0810 22:28:40.692347 (+ 1645us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:40.692356 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:40.692454 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:40.693133 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:40.693136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:40.693153 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:40.693246 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.693252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.694609 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.694612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.695392 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.695401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.697104 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.697110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.697212 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.697234 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:28:40.697304 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 22:28:40.697383 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1601,"mutex_wait_us":927,"thread_start_us":102,"threads_started":1}
I20260810 22:28:40.697662 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.691383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.1.221.130:37831)
0810 22:28:40.692253 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:40.692261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:40.692445 (+ 184us) server_negotiation.cc:407] Connection header received
0810 22:28:40.692592 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:40.692598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:40.692810 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:40.692955 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:40.693372 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.694452 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.695519 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.696369 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.696486 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.696657 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:40.697382 (+ 725us) server_negotiation.cc:299] Negotiation successful
0810 22:28:40.697474 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":100,"threads_started":1}
I20260810 22:28:40.699075 3723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.690901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46543 (local address 127.0.0.1:34062)
0810 22:28:40.691953 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:40.691964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:40.692048 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:40.692526 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:40.692535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:40.692559 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:40.692687 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.692694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.693880 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.693883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.695509 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.695516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.698716 (+ 3200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.698720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.698801 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.698822 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 22:28:40.698884 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 22:28:40.698954 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":94,"thread_start_us":103,"threads_started":1}
I20260810 22:28:40.699163 3720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.691207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34062 (local address 127.1.221.129:46543)
0810 22:28:40.691801 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:40.691808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:40.692061 (+ 253us) server_negotiation.cc:407] Connection header received
0810 22:28:40.692143 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:40.692147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:40.692318 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:40.692488 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:40.692811 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.693763 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.697121 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.698589 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.698724 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.698882 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:40.698931 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 22:28:40.699020 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":68,"threads_started":1}
I20260810 22:28:40.700322 3627 tablet_service.cc:1467] Processing CreateTablet for tablet 352c445350884b489bde0eaf647607fe (DEFAULT_TABLE table=test-table [id=500f8187f78340ab962b466dd92939b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:40.700322 3461 tablet_service.cc:1467] Processing CreateTablet for tablet 352c445350884b489bde0eaf647607fe (DEFAULT_TABLE table=test-table [id=500f8187f78340ab962b466dd92939b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:40.701501 3461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352c445350884b489bde0eaf647607fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:40.701512 3627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352c445350884b489bde0eaf647607fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:40.701705 3305 tablet_service.cc:1467] Processing CreateTablet for tablet 352c445350884b489bde0eaf647607fe (DEFAULT_TABLE table=test-table [id=500f8187f78340ab962b466dd92939b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:40.702587 3305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 352c445350884b489bde0eaf647607fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:40.708587 3724 tablet_bootstrap.cc:492] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a: Bootstrap starting.
I20260810 22:28:40.710762 3726 tablet_bootstrap.cc:492] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766: Bootstrap starting.
I20260810 22:28:40.710762 3725 tablet_bootstrap.cc:492] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be: Bootstrap starting.
I20260810 22:28:40.711190 3724 tablet_bootstrap.cc:654] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:40.711930 3724 log.cc:824] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:40.713001 3726 tablet_bootstrap.cc:654] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:40.713001 3725 tablet_bootstrap.cc:654] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:40.713882 3724 tablet_bootstrap.cc:492] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a: No bootstrap required, opened a new log
I20260810 22:28:40.713898 3725 log.cc:824] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:40.713898 3726 log.cc:824] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:40.714090 3724 ts_tablet_manager.cc:1397] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260810 22:28:40.716176 3725 tablet_bootstrap.cc:492] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be: No bootstrap required, opened a new log
I20260810 22:28:40.716193 3726 tablet_bootstrap.cc:492] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766: No bootstrap required, opened a new log
I20260810 22:28:40.716367 3726 ts_tablet_manager.cc:1397] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:28:40.716367 3725 ts_tablet_manager.cc:1397] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:28:40.718401 3724 raft_consensus.cc:348] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } }
I20260810 22:28:40.718650 3724 raft_consensus.cc:374] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:40.718719 3724 raft_consensus.cc:729] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10958515cd76484292c779df099a7f3a, State: Initialized, Role: FOLLOWER
I20260810 22:28:40.719202 3724 consensus_queue.cc:260] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } }
I20260810 22:28:40.719859 3377 heartbeater.cc:503] Master 127.1.221.190:41619 was elected leader, sending a full tablet report...
I20260810 22:28:40.720227 3724 ts_tablet_manager.cc:1428] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260810 22:28:40.720327 3726 raft_consensus.cc:348] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } }
I20260810 22:28:40.720327 3725 raft_consensus.cc:348] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } }
I20260810 22:28:40.720600 3726 raft_consensus.cc:374] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:40.720600 3725 raft_consensus.cc:374] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:40.720654 3725 raft_consensus.cc:729] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823ac8d5523f4526ba4bd50e7eeeb7be, State: Initialized, Role: FOLLOWER
I20260810 22:28:40.720664 3726 raft_consensus.cc:729] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22c99ccdf344b229edd87a583455766, State: Initialized, Role: FOLLOWER
I20260810 22:28:40.721117 3725 consensus_queue.cc:260] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } }
I20260810 22:28:40.721117 3726 consensus_queue.cc:260] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } }
I20260810 22:28:40.722028 3537 heartbeater.cc:503] Master 127.1.221.190:41619 was elected leader, sending a full tablet report...
I20260810 22:28:40.722151 3726 ts_tablet_manager.cc:1428] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:28:40.722151 3725 ts_tablet_manager.cc:1428] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:28:40.722491 3699 heartbeater.cc:503] Master 127.1.221.190:41619 was elected leader, sending a full tablet report...
I20260810 22:28:40.741551 3730 raft_consensus.cc:482] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:28:40.741783 3730 raft_consensus.cc:504] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } }
I20260810 22:28:40.742913 3730 leader_election.cc:288] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a22c99ccdf344b229edd87a583455766 (127.1.221.131:35433), 823ac8d5523f4526ba4bd50e7eeeb7be (127.1.221.130:37831)
I20260810 22:28:40.747884 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.743304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:47777 (local address 127.1.221.130:37831)
0810 22:28:40.743457 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:40.743463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:40.743778 (+ 315us) server_negotiation.cc:407] Connection header received
0810 22:28:40.743947 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:40.743950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:40.744003 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:40.744083 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:40.744490 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.745350 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.746735 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.747298 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.747431 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.747490 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:40.747669 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 22:28:40.747733 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:28:40.747938 3733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.743237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:37831 (local address 127.1.221.129:47777)
0810 22:28:40.743666 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:40.743682 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:40.743819 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:40.744107 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:40.744110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:40.744159 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:40.744362 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.744370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.745474 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.745482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.746577 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.746590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.747419 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.747426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.747523 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.747554 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:28:40.747636 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 22:28:40.747720 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":101,"spinlock_wait_cycles":49024,"thread_start_us":155,"threads_started":1}
I20260810 22:28:40.748054 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.743393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:36585 (local address 127.1.221.131:35433)
0810 22:28:40.743521 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:40.743526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:40.743825 (+ 299us) server_negotiation.cc:407] Connection header received
0810 22:28:40.743946 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:40.743949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:40.744003 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:40.744083 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:40.744490 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.745367 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.746818 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.747353 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.747478 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.747532 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:40.747827 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 22:28:40.747933 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:28:40.748402 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.743237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:35433 (local address 127.1.221.129:36585)
0810 22:28:40.743705 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:40.743719 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:40.743819 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:40.744107 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:40.744110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:40.744132 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:40.744365 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.744370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.745515 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.745518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.746609 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:40.746622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.747591 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.747596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:40.747697 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.747716 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:28:40.747787 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 22:28:40.747845 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":105,"spinlock_wait_cycles":2816,"thread_start_us":128,"threads_started":1}
I20260810 22:28:40.748749 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352c445350884b489bde0eaf647607fe" candidate_uuid: "10958515cd76484292c779df099a7f3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" is_pre_election: true
I20260810 22:28:40.749076 3648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352c445350884b489bde0eaf647607fe" candidate_uuid: "10958515cd76484292c779df099a7f3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22c99ccdf344b229edd87a583455766" is_pre_election: true
I20260810 22:28:40.749135 3484 raft_consensus.cc:2454] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10958515cd76484292c779df099a7f3a in term 0.
I20260810 22:28:40.749356 3648 raft_consensus.cc:2454] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10958515cd76484292c779df099a7f3a in term 0.
I20260810 22:28:40.749729 3250 leader_election.cc:302] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10958515cd76484292c779df099a7f3a, 823ac8d5523f4526ba4bd50e7eeeb7be; no voters:
I20260810 22:28:40.750118 3730 raft_consensus.cc:2781] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:28:40.750203 3730 raft_consensus.cc:482] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:28:40.750234 3730 raft_consensus.cc:3037] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:40.751394 3730 raft_consensus.cc:504] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } }
I20260810 22:28:40.751813 3730 leader_election.cc:288] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [CANDIDATE]: Term 1 election: Requested vote from peers a22c99ccdf344b229edd87a583455766 (127.1.221.131:35433), 823ac8d5523f4526ba4bd50e7eeeb7be (127.1.221.130:37831)
I20260810 22:28:40.752185 3648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352c445350884b489bde0eaf647607fe" candidate_uuid: "10958515cd76484292c779df099a7f3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a22c99ccdf344b229edd87a583455766"
I20260810 22:28:40.752286 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "352c445350884b489bde0eaf647607fe" candidate_uuid: "10958515cd76484292c779df099a7f3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be"
I20260810 22:28:40.752355 3648 raft_consensus.cc:3037] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:40.752449 3484 raft_consensus.cc:3037] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:40.753535 3648 raft_consensus.cc:2454] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10958515cd76484292c779df099a7f3a in term 1.
I20260810 22:28:40.753945 3251 leader_election.cc:302] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10958515cd76484292c779df099a7f3a, a22c99ccdf344b229edd87a583455766; no voters:
I20260810 22:28:40.754134 3730 raft_consensus.cc:2781] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:40.754333 3484 raft_consensus.cc:2454] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10958515cd76484292c779df099a7f3a in term 1.
I20260810 22:28:40.754366 3730 raft_consensus.cc:686] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [term 1 LEADER]: Becoming Leader. State: Replica: 10958515cd76484292c779df099a7f3a, State: Running, Role: LEADER
I20260810 22:28:40.754766 3730 consensus_queue.cc:237] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } }
I20260810 22:28:40.758113 3149 catalog_manager.cc:5495] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10958515cd76484292c779df099a7f3a (127.1.221.129). New cstate: current_term: 1 leader_uuid: "10958515cd76484292c779df099a7f3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } health_report { overall_health: HEALTHY } } }
I20260810 22:28:40.791446 3720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:40.787834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34070 (local address 127.1.221.129:46543)
0810 22:28:40.787983 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:40.787987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:40.788002 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:28:40.788160 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:40.788162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:40.788221 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:40.788308 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:28:40.788608 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.789124 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.789996 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:40.790213 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:40.790290 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:40.790586 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:28:40.791056 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:28:40.791157 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:40.791219 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 22:28:40.791272 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:28:40.798102 3484 raft_consensus.cc:1261] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be [term 1 FOLLOWER]: Refusing update from remote peer 10958515cd76484292c779df099a7f3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:40.798924 3730 consensus_queue.cc:1035] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:40.799548 3648 raft_consensus.cc:1261] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766 [term 1 FOLLOWER]: Refusing update from remote peer 10958515cd76484292c779df099a7f3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:40.800136 3730 consensus_queue.cc:1035] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:40.808359 3740 mvcc.cc:205] Tried to move back new op lower bound from 7317098171580268544 to 7317098171414335488. Current Snapshot: MvccSnapshot[applied={T|T < 7317098171580268544}]
I20260810 22:28:40.808590 3741 mvcc.cc:205] Tried to move back new op lower bound from 7317098171580268544 to 7317098171414335488. Current Snapshot: MvccSnapshot[applied={T|T < 7317098171580268544}]
I20260810 22:28:40.824519 3150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:28:40.824805 3150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:28:40.827600 3150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1: Sending DeleteTablet for 3 replicas of tablet 352c445350884b489bde0eaf647607fe
I20260810 22:28:40.828548 3627 tablet_service.cc:1514] Processing DeleteTablet for tablet 352c445350884b489bde0eaf647607fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:40 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:52592
I20260810 22:28:40.828542 3461 tablet_service.cc:1514] Processing DeleteTablet for tablet 352c445350884b489bde0eaf647607fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:40 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:33086
I20260810 22:28:40.828851 3305 tablet_service.cc:1514] Processing DeleteTablet for tablet 352c445350884b489bde0eaf647607fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:40 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:34062
I20260810 22:28:40.829187 3749 tablet_replica.cc:330] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766: stopping tablet replica
I20260810 22:28:40.829561 3749 raft_consensus.cc:2229] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:28:40.829947 3749 raft_consensus.cc:2258] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:40.830025 3150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:28:40.830506 3150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:28:40.831491 3750 tablet_replica.cc:330] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be: stopping tablet replica
I20260810 22:28:40.831863 3750 raft_consensus.cc:2229] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:28:40.831916 3749 ts_tablet_manager.cc:1905] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:40.832218 3750 raft_consensus.cc:2258] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:40.832562 3751 tablet_replica.cc:330] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a: stopping tablet replica
I20260810 22:28:40.832901 3751 raft_consensus.cc:2229] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:28:40.833341 3751 raft_consensus.cc:2258] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:40.835460 3749 ts_tablet_manager.cc:1918] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:28:40.835642 3749 log.cc:1196] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/352c445350884b489bde0eaf647607fe
I20260810 22:28:40.836009 3627 tablet_service.cc:1467] Processing CreateTablet for tablet 6885c66d26444acd84e144759d133d85 (DEFAULT_TABLE table=test-table [id=611ea1c4010e4f0390e4fe3e10f6d8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:40.836262 3627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6885c66d26444acd84e144759d133d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:40.836284 3305 tablet_service.cc:1467] Processing CreateTablet for tablet 6885c66d26444acd84e144759d133d85 (DEFAULT_TABLE table=test-table [id=611ea1c4010e4f0390e4fe3e10f6d8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:40.836378 3750 ts_tablet_manager.cc:1905] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:40.836556 3305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6885c66d26444acd84e144759d133d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:40.839216 3750 ts_tablet_manager.cc:1918] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:28:40.839337 3750 log.cc:1196] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/352c445350884b489bde0eaf647607fe
I20260810 22:28:40.839648 3724 tablet_bootstrap.cc:492] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a: Bootstrap starting.
I20260810 22:28:40.839754 3750 ts_tablet_manager.cc:1939] T 352c445350884b489bde0eaf647607fe P 823ac8d5523f4526ba4bd50e7eeeb7be: Deleting consensus metadata
I20260810 22:28:40.840188 3461 tablet_service.cc:1467] Processing CreateTablet for tablet 6885c66d26444acd84e144759d133d85 (DEFAULT_TABLE table=test-table [id=611ea1c4010e4f0390e4fe3e10f6d8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:40.840438 3461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6885c66d26444acd84e144759d133d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:40.840766 3724 tablet_bootstrap.cc:654] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:40.841077 3134 catalog_manager.cc:4841] TS 823ac8d5523f4526ba4bd50e7eeeb7be (127.1.221.130:37831): tablet 352c445350884b489bde0eaf647607fe (table test-table [id=500f8187f78340ab962b466dd92939b9]) successfully deleted
I20260810 22:28:40.841779 3751 ts_tablet_manager.cc:1905] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:40.839716 3726 tablet_bootstrap.cc:492] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766: Bootstrap starting.
I20260810 22:28:40.842847 3726 tablet_bootstrap.cc:654] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:40.843331 3725 tablet_bootstrap.cc:492] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be: Bootstrap starting.
I20260810 22:28:40.839512 3749 ts_tablet_manager.cc:1939] T 352c445350884b489bde0eaf647607fe P a22c99ccdf344b229edd87a583455766: Deleting consensus metadata
I20260810 22:28:40.844412 3725 tablet_bootstrap.cc:654] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:40.844715 3751 ts_tablet_manager.cc:1918] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:28:40.844834 3751 log.cc:1196] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/wal/wals/352c445350884b489bde0eaf647607fe
I20260810 22:28:40.845033 3135 catalog_manager.cc:4841] TS a22c99ccdf344b229edd87a583455766 (127.1.221.131:35433): tablet 352c445350884b489bde0eaf647607fe (table test-table [id=500f8187f78340ab962b466dd92939b9]) successfully deleted
I20260810 22:28:40.845198 3751 ts_tablet_manager.cc:1939] T 352c445350884b489bde0eaf647607fe P 10958515cd76484292c779df099a7f3a: Deleting consensus metadata
I20260810 22:28:40.845918 3724 tablet_bootstrap.cc:492] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a: No bootstrap required, opened a new log
I20260810 22:28:40.845997 3724 ts_tablet_manager.cc:1397] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260810 22:28:40.846140 3726 tablet_bootstrap.cc:492] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766: No bootstrap required, opened a new log
I20260810 22:28:40.846226 3726 ts_tablet_manager.cc:1397] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 22:28:40.846395 3136 catalog_manager.cc:4841] TS 10958515cd76484292c779df099a7f3a (127.1.221.129:46543): tablet 352c445350884b489bde0eaf647607fe (table test-table [id=500f8187f78340ab962b466dd92939b9]) successfully deleted
I20260810 22:28:40.846444 3724 raft_consensus.cc:348] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:40.846558 3724 raft_consensus.cc:374] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:40.846588 3724 raft_consensus.cc:729] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10958515cd76484292c779df099a7f3a, State: Initialized, Role: FOLLOWER
I20260810 22:28:40.846693 3724 consensus_queue.cc:260] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:40.846724 3726 raft_consensus.cc:348] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:40.846851 3726 raft_consensus.cc:374] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:40.846886 3726 raft_consensus.cc:729] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22c99ccdf344b229edd87a583455766, State: Initialized, Role: FOLLOWER
I20260810 22:28:40.846930 3724 ts_tablet_manager.cc:1428] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:28:40.847033 3726 consensus_queue.cc:260] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:40.847303 3726 ts_tablet_manager.cc:1428] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:28:40.847405 3725 tablet_bootstrap.cc:492] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be: No bootstrap required, opened a new log
I20260810 22:28:40.847483 3725 ts_tablet_manager.cc:1397] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 22:28:40.847970 3725 raft_consensus.cc:348] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:40.848086 3725 raft_consensus.cc:374] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:40.848119 3725 raft_consensus.cc:729] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823ac8d5523f4526ba4bd50e7eeeb7be, State: Initialized, Role: FOLLOWER
I20260810 22:28:40.848229 3725 consensus_queue.cc:260] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:40.848501 3725 ts_tablet_manager.cc:1428] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260810 22:28:40.884294 3700 tablet.cc:2369] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:28:40.908483 3538 tablet.cc:2369] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:28:40.920805 3378 tablet.cc:2369] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:28:41.141152 3731 raft_consensus.cc:482] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:28:41.141355 3731 raft_consensus.cc:504] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:41.142518 3731 leader_election.cc:288] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10958515cd76484292c779df099a7f3a (127.1.221.129:46543), 823ac8d5523f4526ba4bd50e7eeeb7be (127.1.221.130:37831)
I20260810 22:28:41.146977 3720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:41.143020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:39885 (local address 127.1.221.129:46543)
0810 22:28:41.143186 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:41.143192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:41.143396 (+ 204us) server_negotiation.cc:407] Connection header received
0810 22:28:41.143471 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:41.143477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:41.143531 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:41.143592 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:41.143937 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.144558 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.145961 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.146385 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.146512 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:41.146571 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:41.146742 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 22:28:41.146834 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 22:28:41.147032 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:41.142926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:44027 (local address 127.1.221.130:37831)
0810 22:28:41.143091 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:41.143099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:41.143351 (+ 252us) server_negotiation.cc:407] Connection header received
0810 22:28:41.143498 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:41.143501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:41.143547 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:41.143636 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:41.144003 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.144557 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.145995 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.146388 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.146517 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:41.146572 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:41.146795 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 22:28:41.146881 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 22:28:41.147051 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:41.142805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:37831 (local address 127.1.221.131:44027)
0810 22:28:41.143246 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:41.143258 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:41.143393 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:41.143673 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:41.143676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:41.143695 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:41.143849 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:41.143854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.144679 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.144682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:41.145859 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:41.145872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.146539 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.146543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:41.146650 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:41.146668 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 22:28:41.146762 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 22:28:41.146877 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":190,"thread_start_us":120,"threads_started":1}
I20260810 22:28:41.147039 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:41.142805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:46543 (local address 127.1.221.131:39885)
0810 22:28:41.143235 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:41.143248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:41.143367 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:41.143631 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:41.143634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:41.143650 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:41.143825 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:41.143832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.144671 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.144676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:41.145833 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:41.145844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.146506 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.146510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:41.146610 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:41.146641 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:28:41.146704 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 22:28:41.146848 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":160,"thread_start_us":85,"threads_started":1}
I20260810 22:28:41.147678 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6885c66d26444acd84e144759d133d85" candidate_uuid: "a22c99ccdf344b229edd87a583455766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" is_pre_election: true
I20260810 22:28:41.147855 3484 raft_consensus.cc:2454] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22c99ccdf344b229edd87a583455766 in term 0.
I20260810 22:28:41.148218 3563 leader_election.cc:302] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 823ac8d5523f4526ba4bd50e7eeeb7be, a22c99ccdf344b229edd87a583455766; no voters:
I20260810 22:28:41.148222 3328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6885c66d26444acd84e144759d133d85" candidate_uuid: "a22c99ccdf344b229edd87a583455766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10958515cd76484292c779df099a7f3a" is_pre_election: true
I20260810 22:28:41.148464 3328 raft_consensus.cc:2454] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22c99ccdf344b229edd87a583455766 in term 0.
I20260810 22:28:41.148504 3731 raft_consensus.cc:2781] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:28:41.148573 3731 raft_consensus.cc:482] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:28:41.148629 3731 raft_consensus.cc:3037] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:41.149772 3731 raft_consensus.cc:504] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:41.150153 3731 leader_election.cc:288] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [CANDIDATE]: Term 1 election: Requested vote from peers 10958515cd76484292c779df099a7f3a (127.1.221.129:46543), 823ac8d5523f4526ba4bd50e7eeeb7be (127.1.221.130:37831)
I20260810 22:28:41.150492 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6885c66d26444acd84e144759d133d85" candidate_uuid: "a22c99ccdf344b229edd87a583455766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be"
I20260810 22:28:41.150516 3328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6885c66d26444acd84e144759d133d85" candidate_uuid: "a22c99ccdf344b229edd87a583455766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10958515cd76484292c779df099a7f3a"
I20260810 22:28:41.150624 3484 raft_consensus.cc:3037] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:41.150636 3328 raft_consensus.cc:3037] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:41.151561 3484 raft_consensus.cc:2454] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22c99ccdf344b229edd87a583455766 in term 1.
I20260810 22:28:41.151654 3328 raft_consensus.cc:2454] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22c99ccdf344b229edd87a583455766 in term 1.
I20260810 22:28:41.151850 3563 leader_election.cc:302] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 823ac8d5523f4526ba4bd50e7eeeb7be, a22c99ccdf344b229edd87a583455766; no voters:
I20260810 22:28:41.152037 3731 raft_consensus.cc:2781] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:41.152246 3731 raft_consensus.cc:686] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [term 1 LEADER]: Becoming Leader. State: Replica: a22c99ccdf344b229edd87a583455766, State: Running, Role: LEADER
I20260810 22:28:41.152514 3731 consensus_queue.cc:237] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:41.154877 3150 catalog_manager.cc:5495] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 reported cstate change: term changed from 0 to 1, leader changed from <none> to a22c99ccdf344b229edd87a583455766 (127.1.221.131). New cstate: current_term: 1 leader_uuid: "a22c99ccdf344b229edd87a583455766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } health_report { overall_health: HEALTHY } } }
Aug 10 22:28:41 dist-test-slave-njxd krb5kdc[3102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400919, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:41.259791 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:41.251326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41619 (local address 127.0.0.1:36602)
0810 22:28:41.251647 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:41.251663 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:41.251824 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:41.252077 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:41.252080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:41.252520 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:41.252615 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:41.252621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.253386 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.253391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:41.254012 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:41.254020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.254468 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.254471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:41.254557 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:41.255106 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:41.255118 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:41.256602 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:41.258250 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:41.258260 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:41.258266 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:41.258525 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:41.258808 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:41.258810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:41.258812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:41.258895 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:41.259245 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:41.259251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:41.259366 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 22:28:41.259460 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 22:28:41.259578 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":106,"threads_started":1}
I20260810 22:28:41.259829 3769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:41.251361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.1.221.190:41619)
0810 22:28:41.251627 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:41.251636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:41.251747 (+ 111us) server_negotiation.cc:407] Connection header received
0810 22:28:41.251931 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:41.251934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:41.251969 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:41.252060 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:41.252733 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.253281 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.254125 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.254358 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.254464 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:41.256742 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:41.256745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:41.256747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:41.256769 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:41.258141 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:41.258643 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:41.258646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:41.258647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:41.258728 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:41.258971 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:41.258974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:41.258975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:41.259146 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:41.259221 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:41.259514 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 22:28:41.259688 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":207,"thread_start_us":127,"threads_started":1}
I20260810 22:28:41.264046 3150 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:36602:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 22:28:41.264925 3150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:28:41.270016 3461 tablet_service.cc:1467] Processing CreateTablet for tablet 573fa3919dc54db696b6c919335b323a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7046a346a57f4e9fbe5bca254e9e5106]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:28:41.269950 3305 tablet_service.cc:1467] Processing CreateTablet for tablet 573fa3919dc54db696b6c919335b323a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7046a346a57f4e9fbe5bca254e9e5106]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:28:41.270211 3627 tablet_service.cc:1467] Processing CreateTablet for tablet 573fa3919dc54db696b6c919335b323a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7046a346a57f4e9fbe5bca254e9e5106]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:28:41.270496 3461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 573fa3919dc54db696b6c919335b323a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:41.270498 3305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 573fa3919dc54db696b6c919335b323a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:41.270569 3627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 573fa3919dc54db696b6c919335b323a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:41.273094 3725 tablet_bootstrap.cc:492] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be: Bootstrap starting.
I20260810 22:28:41.273399 3726 tablet_bootstrap.cc:492] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766: Bootstrap starting.
I20260810 22:28:41.274183 3725 tablet_bootstrap.cc:654] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:41.274456 3726 tablet_bootstrap.cc:654] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:41.274976 3724 tablet_bootstrap.cc:492] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a: Bootstrap starting.
I20260810 22:28:41.275585 3725 tablet_bootstrap.cc:492] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be: No bootstrap required, opened a new log
I20260810 22:28:41.275681 3725 ts_tablet_manager.cc:1397] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 22:28:41.275879 3724 tablet_bootstrap.cc:654] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:41.276212 3725 raft_consensus.cc:348] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:41.276355 3725 raft_consensus.cc:374] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:41.276403 3725 raft_consensus.cc:729] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823ac8d5523f4526ba4bd50e7eeeb7be, State: Initialized, Role: FOLLOWER
I20260810 22:28:41.276575 3725 consensus_queue.cc:260] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:41.276912 3732 tablet_replica.cc:439] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } }
I20260810 22:28:41.277078 3732 tablet_replica.cc:442] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:41.277206 3726 tablet_bootstrap.cc:492] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766: No bootstrap required, opened a new log
I20260810 22:28:41.277297 3726 ts_tablet_manager.cc:1397] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 22:28:41.277756 3725 ts_tablet_manager.cc:1428] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 22:28:41.277786 3726 raft_consensus.cc:348] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:41.277889 3726 raft_consensus.cc:374] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:41.277921 3726 raft_consensus.cc:729] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22c99ccdf344b229edd87a583455766, State: Initialized, Role: FOLLOWER
I20260810 22:28:41.278086 3724 tablet_bootstrap.cc:492] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a: No bootstrap required, opened a new log
I20260810 22:28:41.278056 3726 consensus_queue.cc:260] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:41.278182 3724 ts_tablet_manager.cc:1397] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 22:28:41.278299 3731 tablet_replica.cc:439] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } }
I20260810 22:28:41.278422 3731 tablet_replica.cc:442] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:41.278657 3726 ts_tablet_manager.cc:1428] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:28:41.278672 3724 raft_consensus.cc:348] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:41.278796 3724 raft_consensus.cc:374] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:41.278833 3724 raft_consensus.cc:729] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10958515cd76484292c779df099a7f3a, State: Initialized, Role: FOLLOWER
I20260810 22:28:41.278995 3724 consensus_queue.cc:260] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:41.279282 3724 ts_tablet_manager.cc:1428] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:28:41.279515 3730 tablet_replica.cc:439] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } }
I20260810 22:28:41.279659 3730 tablet_replica.cc:442] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:41.359458 3731 raft_consensus.cc:482] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:28:41.359601 3731 raft_consensus.cc:504] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:41.360013 3731 leader_election.cc:288] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10958515cd76484292c779df099a7f3a (127.1.221.129:46543), 823ac8d5523f4526ba4bd50e7eeeb7be (127.1.221.130:37831)
I20260810 22:28:41.360504 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573fa3919dc54db696b6c919335b323a" candidate_uuid: "a22c99ccdf344b229edd87a583455766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" is_pre_election: true
I20260810 22:28:41.360527 3328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573fa3919dc54db696b6c919335b323a" candidate_uuid: "a22c99ccdf344b229edd87a583455766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10958515cd76484292c779df099a7f3a" is_pre_election: true
I20260810 22:28:41.360677 3484 raft_consensus.cc:2454] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22c99ccdf344b229edd87a583455766 in term 0.
I20260810 22:28:41.360752 3328 raft_consensus.cc:2454] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22c99ccdf344b229edd87a583455766 in term 0.
I20260810 22:28:41.361028 3563 leader_election.cc:302] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 823ac8d5523f4526ba4bd50e7eeeb7be, a22c99ccdf344b229edd87a583455766; no voters:
I20260810 22:28:41.361219 3731 raft_consensus.cc:2781] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:28:41.361294 3731 raft_consensus.cc:482] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:28:41.361347 3731 raft_consensus.cc:3037] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:41.362509 3731 raft_consensus.cc:504] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:41.362860 3731 leader_election.cc:288] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [CANDIDATE]: Term 1 election: Requested vote from peers 10958515cd76484292c779df099a7f3a (127.1.221.129:46543), 823ac8d5523f4526ba4bd50e7eeeb7be (127.1.221.130:37831)
I20260810 22:28:41.363184 3328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573fa3919dc54db696b6c919335b323a" candidate_uuid: "a22c99ccdf344b229edd87a583455766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10958515cd76484292c779df099a7f3a"
I20260810 22:28:41.363214 3484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "573fa3919dc54db696b6c919335b323a" candidate_uuid: "a22c99ccdf344b229edd87a583455766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be"
I20260810 22:28:41.363322 3328 raft_consensus.cc:3037] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:41.363345 3484 raft_consensus.cc:3037] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:41.364324 3484 raft_consensus.cc:2454] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22c99ccdf344b229edd87a583455766 in term 1.
I20260810 22:28:41.364333 3328 raft_consensus.cc:2454] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22c99ccdf344b229edd87a583455766 in term 1.
I20260810 22:28:41.364639 3563 leader_election.cc:302] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 823ac8d5523f4526ba4bd50e7eeeb7be, a22c99ccdf344b229edd87a583455766; no voters:
I20260810 22:28:41.364821 3731 raft_consensus.cc:2781] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:41.364923 3731 raft_consensus.cc:686] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [term 1 LEADER]: Becoming Leader. State: Replica: a22c99ccdf344b229edd87a583455766, State: Running, Role: LEADER
I20260810 22:28:41.365056 3731 consensus_queue.cc:237] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } }
I20260810 22:28:41.365811 3758 tablet_replica.cc:439] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766: TxnStatusTablet state changed. Reason: New leader a22c99ccdf344b229edd87a583455766. Latest consensus state: current_term: 1 leader_uuid: "a22c99ccdf344b229edd87a583455766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } }
I20260810 22:28:41.365916 3758 tablet_replica.cc:442] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:28:41.366304 3775 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 22:28:41.366511 3775 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 22:28:41.366901 3147 catalog_manager.cc:5495] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 reported cstate change: term changed from 0 to 1, leader changed from <none> to a22c99ccdf344b229edd87a583455766 (127.1.221.131). New cstate: current_term: 1 leader_uuid: "a22c99ccdf344b229edd87a583455766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } health_report { overall_health: HEALTHY } } }
I20260810 22:28:41.380815 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:41.376592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:35433 (local address 127.0.0.1:52600)
0810 22:28:41.376807 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:41.376825 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:41.376935 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:41.377233 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:41.377236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:41.377245 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:28:41.377361 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:41.377368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.378224 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.378228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:41.379208 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:41.379219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.379698 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.379702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:41.379810 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:41.379875 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:28:41.380483 (+ 608us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:28:41.380498 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 22:28:41.380571 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 22:28:41.380663 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 22:28:41.380887 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:41.376740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.1.221.131:35433)
0810 22:28:41.376964 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:41.376968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:41.376982 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:28:41.377047 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:41.377049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:41.377097 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:41.377193 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:28:41.377460 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.378097 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.379324 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.379583 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.379668 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:41.380004 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:28:41.380395 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:28:41.380478 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:41.380598 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 22:28:41.380700 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260810 22:28:41.383908 3484 raft_consensus.cc:1261] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be [term 1 FOLLOWER]: Refusing update from remote peer a22c99ccdf344b229edd87a583455766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:41.384104 3328 raft_consensus.cc:1261] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a [term 1 FOLLOWER]: Refusing update from remote peer a22c99ccdf344b229edd87a583455766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:41.384389 3731 consensus_queue.cc:1035] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:41.384596 3758 consensus_queue.cc:1035] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:41.385993 3732 tablet_replica.cc:439] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be: TxnStatusTablet state changed. Reason: New leader a22c99ccdf344b229edd87a583455766. Latest consensus state: current_term: 1 leader_uuid: "a22c99ccdf344b229edd87a583455766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } }
I20260810 22:28:41.386102 3732 tablet_replica.cc:442] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:41.386935 3730 tablet_replica.cc:439] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a: TxnStatusTablet state changed. Reason: New leader a22c99ccdf344b229edd87a583455766. Latest consensus state: current_term: 1 leader_uuid: "a22c99ccdf344b229edd87a583455766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } }
I20260810 22:28:41.387056 3730 tablet_replica.cc:442] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:41.389063 3758 tablet_replica.cc:439] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a22c99ccdf344b229edd87a583455766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } }
I20260810 22:28:41.389183 3758 tablet_replica.cc:442] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:28:41.389299 3758 tablet_replica.cc:439] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a22c99ccdf344b229edd87a583455766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } }
I20260810 22:28:41.389366 3758 tablet_replica.cc:442] T 573fa3919dc54db696b6c919335b323a P a22c99ccdf344b229edd87a583455766: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:28:41.389703 3730 tablet_replica.cc:439] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a22c99ccdf344b229edd87a583455766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } }
I20260810 22:28:41.389827 3730 tablet_replica.cc:442] T 573fa3919dc54db696b6c919335b323a P 10958515cd76484292c779df099a7f3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:41.389859 3732 tablet_replica.cc:439] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a22c99ccdf344b229edd87a583455766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 } } peers { permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 } } peers { permanent_uuid: "a22c99ccdf344b229edd87a583455766" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35433 } } }
I20260810 22:28:41.389976 3732 tablet_replica.cc:442] T 573fa3919dc54db696b6c919335b323a P 823ac8d5523f4526ba4bd50e7eeeb7be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:41.390638 3782 mvcc.cc:205] Tried to move back new op lower bound from 7317098173983551488 to 7317098173912776704. Current Snapshot: MvccSnapshot[applied={T|T < 7317098173983551488}]
I20260810 22:28:41.399863 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:41.396277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.1.221.131:35433)
0810 22:28:41.396391 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:41.396395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:41.396536 (+ 141us) server_negotiation.cc:407] Connection header received
0810 22:28:41.396649 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:41.396652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:41.396701 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:41.396784 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:28:41.397001 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.397842 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.398782 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.399124 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.399239 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:41.399438 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:28:41.399521 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:28:41.399587 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:41.399672 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 22:28:41.399728 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 22:28:41.412627 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:41.408563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:50581 (local address 127.1.221.131:35433)
0810 22:28:41.408666 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:41.408670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:41.408685 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:28:41.408780 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:41.408783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:41.408831 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:41.408908 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:28:41.409220 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.410060 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.411334 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.411678 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.411840 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:41.412150 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:28:41.412232 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:28:41.412305 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:41.412399 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 22:28:41.412462 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":36}
I20260810 22:28:41.412935 3791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:41.408316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:35433 (local address 127.1.221.131:50581)
0810 22:28:41.408590 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:41.408605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:41.408700 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:41.408931 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:41.408933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:41.408941 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:28:41.409072 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:41.409078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.410189 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.410193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:41.411212 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:41.411222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.411840 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.411844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:41.411945 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:41.412006 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:28:41.412319 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:28:41.412325 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:28:41.412387 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 22:28:41.412469 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":91,"threads_started":1}
I20260810 22:28:41.420269 3484 raft_consensus.cc:1261] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be [term 1 FOLLOWER]: Refusing update from remote peer a22c99ccdf344b229edd87a583455766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:41.420322 3328 raft_consensus.cc:1261] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a [term 1 FOLLOWER]: Refusing update from remote peer a22c99ccdf344b229edd87a583455766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:41.420734 3758 consensus_queue.cc:1035] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "823ac8d5523f4526ba4bd50e7eeeb7be" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:41.420888 3731 consensus_queue.cc:1035] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10958515cd76484292c779df099a7f3a" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 46543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:41.465276 3149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:28:41.465468 3149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:28:41.467739 3149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abe1122d7dae4acc9034e4e48838d5e1: Sending DeleteTablet for 3 replicas of tablet 6885c66d26444acd84e144759d133d85
I20260810 22:28:41.468402 3305 tablet_service.cc:1514] Processing DeleteTablet for tablet 6885c66d26444acd84e144759d133d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:41 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:34062
I20260810 22:28:41.468524 3627 tablet_service.cc:1514] Processing DeleteTablet for tablet 6885c66d26444acd84e144759d133d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:41 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:52592
I20260810 22:28:41.468834 3461 tablet_service.cc:1514] Processing DeleteTablet for tablet 6885c66d26444acd84e144759d133d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:41 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:33086
I20260810 22:28:41.469049 3798 tablet_replica.cc:330] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766: stopping tablet replica
I20260810 22:28:41.469203 3798 raft_consensus.cc:2229] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:28:41.469375 3799 tablet_replica.cc:330] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be: stopping tablet replica
I20260810 22:28:41.469544 3799 raft_consensus.cc:2229] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:28:41.469691 3798 raft_consensus.cc:2258] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:41.469735 3799 raft_consensus.cc:2258] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:41.470059 3797 tablet_replica.cc:330] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a: stopping tablet replica
I20260810 22:28:41.470568 3798 ts_tablet_manager.cc:1905] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:41.470718 3799 ts_tablet_manager.cc:1905] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:41.473760 3798 ts_tablet_manager.cc:1918] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:28:41.473762 3799 ts_tablet_manager.cc:1918] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:28:41.473855 3798 log.cc:1196] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/6885c66d26444acd84e144759d133d85
I20260810 22:28:41.473779 3797 raft_consensus.cc:2229] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:28:41.473960 3799 log.cc:1196] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/6885c66d26444acd84e144759d133d85
I20260810 22:28:41.474053 3797 raft_consensus.cc:2258] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:41.474160 3798 ts_tablet_manager.cc:1939] T 6885c66d26444acd84e144759d133d85 P a22c99ccdf344b229edd87a583455766: Deleting consensus metadata
I20260810 22:28:41.474337 3799 ts_tablet_manager.cc:1939] T 6885c66d26444acd84e144759d133d85 P 823ac8d5523f4526ba4bd50e7eeeb7be: Deleting consensus metadata
I20260810 22:28:41.474943 3797 ts_tablet_manager.cc:1905] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:41.475147 3135 catalog_manager.cc:4841] TS a22c99ccdf344b229edd87a583455766 (127.1.221.131:35433): tablet 6885c66d26444acd84e144759d133d85 (table test-table [id=611ea1c4010e4f0390e4fe3e10f6d8ae]) successfully deleted
I20260810 22:28:41.475147 3134 catalog_manager.cc:4841] TS 823ac8d5523f4526ba4bd50e7eeeb7be (127.1.221.130:37831): tablet 6885c66d26444acd84e144759d133d85 (table test-table [id=611ea1c4010e4f0390e4fe3e10f6d8ae]) successfully deleted
I20260810 22:28:41.477934 3797 ts_tablet_manager.cc:1918] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:28:41.478029 3797 log.cc:1196] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786400913154531-1910-0/minicluster-data/ts-0/wal/wals/6885c66d26444acd84e144759d133d85
I20260810 22:28:41.478309 3797 ts_tablet_manager.cc:1939] T 6885c66d26444acd84e144759d133d85 P 10958515cd76484292c779df099a7f3a: Deleting consensus metadata
I20260810 22:28:41.479036 3136 catalog_manager.cc:4841] TS 10958515cd76484292c779df099a7f3a (127.1.221.129:46543): tablet 6885c66d26444acd84e144759d133d85 (table test-table [id=611ea1c4010e4f0390e4fe3e10f6d8ae]) successfully deleted
Aug 10 22:28:41 dist-test-slave-njxd krb5kdc[3102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400920, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260810 22:28:41.489723 3720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:41.478102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.1.221.129:46543)
0810 22:28:41.478239 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:41.478245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:41.478660 (+ 415us) server_negotiation.cc:407] Connection header received
0810 22:28:41.478829 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:41.478832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:41.478903 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:41.478980 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:41.480047 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.480555 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.481676 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.481899 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.481978 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:41.485984 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:41.485997 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:41.486013 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:41.486041 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:41.487737 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:41.488207 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:41.488219 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:41.488220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:41.488307 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:41.488594 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:41.488598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:41.488600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:41.488882 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:41.489105 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:41.489282 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 22:28:41.489419 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 22:28:41.490285 3371 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40556
I20260810 22:28:41.500248 3769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:41.492164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.1.221.190:41619)
0810 22:28:41.492302 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:41.492306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:41.492339 (+ 33us) server_negotiation.cc:407] Connection header received
0810 22:28:41.492488 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:41.492490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:41.492533 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:41.492592 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:41.493383 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.494001 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.495112 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:41.495342 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:41.495419 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:41.496989 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:41.496992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:41.496995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:41.497020 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:41.498516 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:41.499016 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:41.499019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:41.499020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:41.499101 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:41.499369 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:41.499371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:41.499372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:41.499544 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:41.499669 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:41.499951 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 22:28:41.500064 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 22:28:41.500716 3149 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46620
I20260810 22:28:41.502169 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 3231
I20260810 22:28:41.512163 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 3386
I20260810 22:28:41.522233 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 3546
I20260810 22:28:41.531672 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 3115
2026-08-10T22:28:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3098 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:28:41 dist-test-slave-njxd krb5kdc[3806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:28:41 dist-test-slave-njxd krb5kdc[3806](info): set up 2 sockets
Aug 10 22:28:41 dist-test-slave-njxd krb5kdc[3806](info): commencing operation
krb5kdc: starting...
W20260810 22:28:42.637984 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:28:42 dist-test-slave-njxd krb5kdc[3806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:28:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:28:42.764534 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:37029
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:37029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:42.875427 3819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:42.875726 3819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:42.875800 3819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:42.879782 3819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:28:42.879880 3819 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 22:28:42.879958 3819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:42.880023 3819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:42.880065 3819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:28:42.880101 3819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:28:42.884999 3819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:37029
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:37029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:42.886018 3819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:42.886960 3819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:42.895081 3824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:42.895081 3828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:42.895253 3819 server_base.cc:1034] running on GCE node
W20260810 22:28:42.895104 3825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:42.896028 3819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:42.897042 3819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:42.898228 3819 hybrid_clock.cc:648] HybridClock initialized: now 1786400922898202 us; error 47 us; skew 500 ppm
Aug 10 22:28:42 dist-test-slave-njxd krb5kdc[3806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400922, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:42.901033 3819 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:28:42.901926 3819 webserver.cc:460] Webserver started at http://127.1.221.190:45887/ using document root <none> and password file <none>
I20260810 22:28:42.902475 3819 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:42.902565 3819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:42.902782 3819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:42.904703 3819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "5290a0e324214d519b2655124f48d94d"
format_stamp: "Formatted at 2026-08-10 22:28:42 on dist-test-slave-njxd"
server_key: "4181ab639494a29ae0623282caedc38e"
server_key_iv: "af5aea71b09047a97f2b84e9dc6bb0da"
server_key_version: "encryptionkey@0"
I20260810 22:28:42.905256 3819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "5290a0e324214d519b2655124f48d94d"
format_stamp: "Formatted at 2026-08-10 22:28:42 on dist-test-slave-njxd"
server_key: "4181ab639494a29ae0623282caedc38e"
server_key_iv: "af5aea71b09047a97f2b84e9dc6bb0da"
server_key_version: "encryptionkey@0"
I20260810 22:28:42.908840 3819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 22:28:42.911103 3836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:42.912262 3819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:28:42.912395 3819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "5290a0e324214d519b2655124f48d94d"
format_stamp: "Formatted at 2026-08-10 22:28:42 on dist-test-slave-njxd"
server_key: "4181ab639494a29ae0623282caedc38e"
server_key_iv: "af5aea71b09047a97f2b84e9dc6bb0da"
server_key_version: "encryptionkey@0"
I20260810 22:28:42.912498 3819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:42.928448 3819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:42.932053 3819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:42.932266 3819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:42.940446 3819 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:37029
I20260810 22:28:42.940474 3905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:37029 every 8 connection(s)
I20260810 22:28:42.941383 3819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:28:42.944185 3906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:42.949671 3906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d: Bootstrap starting.
I20260810 22:28:42.950763 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 3819
I20260810 22:28:42.950896 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:28:42.951195 1910 external_mini_cluster.cc:1428] Setting key 6bab8149bebe88b0ca4818a8e0c7e9a4
I20260810 22:28:42.952716 3906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:42.953464 3906 log.cc:824] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:42.955363 3906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d: No bootstrap required, opened a new log
I20260810 22:28:42.959010 3906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290a0e324214d519b2655124f48d94d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37029 } }
I20260810 22:28:42.959266 3906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:42.959339 3906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5290a0e324214d519b2655124f48d94d, State: Initialized, Role: FOLLOWER
Aug 10 22:28:42 dist-test-slave-njxd krb5kdc[3806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:42.959853 3906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290a0e324214d519b2655124f48d94d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37029 } }
I20260810 22:28:42.960027 3906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:28:42.960101 3906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:28:42.960213 3906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:42.961333 3906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290a0e324214d519b2655124f48d94d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37029 } }
I20260810 22:28:42.961762 3906 leader_election.cc:302] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5290a0e324214d519b2655124f48d94d; no voters:
I20260810 22:28:42.962392 3906 leader_election.cc:288] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:28:42.962536 3911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:42.962761 3911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [term 1 LEADER]: Becoming Leader. State: Replica: 5290a0e324214d519b2655124f48d94d, State: Running, Role: LEADER
I20260810 22:28:42.963529 3911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290a0e324214d519b2655124f48d94d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37029 } }
I20260810 22:28:42.963694 3906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:28:42.964780 3911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5290a0e324214d519b2655124f48d94d. Latest consensus state: current_term: 1 leader_uuid: "5290a0e324214d519b2655124f48d94d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290a0e324214d519b2655124f48d94d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37029 } } }
I20260810 22:28:42.964915 3911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:42.965195 3912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5290a0e324214d519b2655124f48d94d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290a0e324214d519b2655124f48d94d" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37029 } } }
I20260810 22:28:42.965296 3912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:42.966763 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:42.952367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.1.221.190:37029)
0810 22:28:42.952950 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:42.952967 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:42.953002 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:28:42.953697 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:42.953717 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:42.954050 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:42.954392 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:42.955213 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:42.956214 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:42.957072 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:42.957548 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:42.957783 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:42.960517 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:42.960522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:42.960535 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:42.960561 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:42.962730 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:42.963307 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:42.963313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:42.963319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:42.963446 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:42.963723 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:42.963726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:42.963728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:42.964194 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:42.964379 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:42.965241 (+ 862us) server_negotiation.cc:299] Negotiation successful
0810 22:28:42.965672 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":209,"threads_started":1}
W20260810 22:28:42.969010 3929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:28:42.969089 3929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:28:42.969148 3922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:28:42.970072 3922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:28:42.975069 3922 catalog_manager.cc:1347] Generated new cluster ID: 4c34d7f992b84349aca32d0d71d9fc5f
I20260810 22:28:42.975134 3922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:28:42.988730 3922 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:28:42.989683 3922 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:28:42.993883 3922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d: Generated new TSK 0
I20260810 22:28:42.994675 3922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:28:43.032931 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37029
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:43.166842 3935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:43.167114 3935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:43.167162 3935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:43.171202 3935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:43.171267 3935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:43.171309 3935 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 22:28:43.171324 3935 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:28:43.171356 3935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:28:43.176730 3935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:37029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:43.177801 3935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:43.178823 3935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:43.188760 3940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:43.188766 3941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:43.188781 3943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:43.189138 3935 server_base.cc:1034] running on GCE node
I20260810 22:28:43.189821 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:43.190544 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:43.191733 3935 hybrid_clock.cc:648] HybridClock initialized: now 1786400923191719 us; error 27 us; skew 500 ppm
Aug 10 22:28:43 dist-test-slave-njxd krb5kdc[3806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:43.194875 3935 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:28:43.195737 3935 webserver.cc:460] Webserver started at http://127.1.221.129:37387/ using document root <none> and password file <none>
I20260810 22:28:43.196286 3935 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:43.196346 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:43.196527 3935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:43.198637 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "880c64863b3a4e5d850578fb56f35bd8"
format_stamp: "Formatted at 2026-08-10 22:28:43 on dist-test-slave-njxd"
server_key: "2a05b396e782f0ed86a73b788aa93402"
server_key_iv: "0b9dd45cde40562f67a6d5f3d3969271"
server_key_version: "encryptionkey@0"
I20260810 22:28:43.199174 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "880c64863b3a4e5d850578fb56f35bd8"
format_stamp: "Formatted at 2026-08-10 22:28:43 on dist-test-slave-njxd"
server_key: "2a05b396e782f0ed86a73b788aa93402"
server_key_iv: "0b9dd45cde40562f67a6d5f3d3969271"
server_key_version: "encryptionkey@0"
I20260810 22:28:43.202921 3935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:28:43.205224 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:43.206476 3935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:28:43.206569 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "880c64863b3a4e5d850578fb56f35bd8"
format_stamp: "Formatted at 2026-08-10 22:28:43 on dist-test-slave-njxd"
server_key: "2a05b396e782f0ed86a73b788aa93402"
server_key_iv: "0b9dd45cde40562f67a6d5f3d3969271"
server_key_version: "encryptionkey@0"
I20260810 22:28:43.206653 3935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:43.223737 3935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:43.227341 3935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:43.227506 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:43.229009 3935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:43.229063 3935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:43.229104 3935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:43.229153 3935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:28:43 dist-test-slave-njxd krb5kdc[3806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:43.240715 3935 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:36241
I20260810 22:28:43.240785 4080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:36241 every 8 connection(s)
I20260810 22:28:43.241835 3935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:28:43.243871 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.229779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:53119 (local address 127.1.221.190:37029)
0810 22:28:43.229936 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:43.229941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:43.230913 (+ 972us) server_negotiation.cc:407] Connection header received
0810 22:28:43.232267 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:43.232273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:43.232338 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:43.232453 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:43.233609 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.234316 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.235804 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.236052 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.236167 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.239909 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:43.239913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:43.239915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:43.239939 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:43.241676 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.242276 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.242279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.242281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.242391 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.242820 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.242824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.242825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.243048 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:43.243135 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:43.243584 (+ 449us) server_negotiation.cc:299] Negotiation successful
0810 22:28:43.243718 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 22:28:43.244679 3961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.230232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37029 (local address 127.1.221.129:53119)
0810 22:28:43.230728 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:43.230789 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:43.232008 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:43.232645 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:43.232653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:43.232965 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:43.233430 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.233443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.234469 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.234473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.235678 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.235688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.236222 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.236226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.236435 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.237132 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:43.237159 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:43.239718 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:43.241809 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.241817 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.241832 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.242157 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.242504 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.242507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.242509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.242704 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.243149 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:43.243154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:43.243370 (+ 216us) client_negotiation.cc:769] Sending connection context
0810 22:28:43.243533 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 22:28:43.243807 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":114,"threads_started":1}
I20260810 22:28:43.246161 4081 heartbeater.cc:347] Connected to a master server at 127.1.221.190:37029
I20260810 22:28:43.246353 4081 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:43.246897 4081 heartbeater.cc:511] Master 127.1.221.190:37029 requested a full tablet report, sending...
I20260810 22:28:43.248425 3857 ts_manager.cc:194] Registered new tserver with Master: 880c64863b3a4e5d850578fb56f35bd8 (127.1.221.129:36241)
I20260810 22:28:43.249843 3857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:53119
I20260810 22:28:43.251255 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 3935
I20260810 22:28:43.251379 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:28:43.251726 1910 external_mini_cluster.cc:1428] Setting key 002f99bccda8dac7ac8d1152a0831e28
I20260810 22:28:43.255849 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.248314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:37963 (local address 127.1.221.190:37029)
0810 22:28:43.248436 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:43.248440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:43.248597 (+ 157us) server_negotiation.cc:407] Connection header received
0810 22:28:43.248759 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:43.248762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:43.248808 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:43.248873 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:43.249395 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.249997 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.250909 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.251247 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.251356 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.252608 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:43.252612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:43.252614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:43.252636 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:43.254177 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.254624 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.254627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.254629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.254707 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.255027 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.255033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.255036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.255253 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:43.255338 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:43.255639 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 22:28:43.255732 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:28:43.255988 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.248200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37029 (local address 127.1.221.129:37963)
0810 22:28:43.248515 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:43.248531 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:43.248663 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:43.248917 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:43.248920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:43.249158 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:43.249277 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.249284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.250143 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.250147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.250797 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.250806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.251383 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.251390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.251487 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.252016 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:43.252031 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:43.252487 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:43.254292 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.254297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.254300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.254498 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.254801 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.254804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.254806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.254895 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.255356 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:43.255360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:43.255470 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 22:28:43.255589 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 22:28:43.255759 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:28:43.280681 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37029
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:43.399982 4091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:43.400317 4091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:43.400419 4091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:43.404532 4091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:43.404631 4091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:43.404702 4091 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 22:28:43.404744 4091 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:28:43.404810 4091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:28:43.410326 4091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:37029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:43.411495 4091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:43.412580 4091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:43.422490 4099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:43.422493 4096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:43.422490 4097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:43.422715 4091 server_base.cc:1034] running on GCE node
I20260810 22:28:43.423278 4091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:43.423971 4091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:43.425137 4091 hybrid_clock.cc:648] HybridClock initialized: now 1786400923425122 us; error 28 us; skew 500 ppm
Aug 10 22:28:43 dist-test-slave-njxd krb5kdc[3806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:43.428141 4091 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:28:43.429040 4091 webserver.cc:460] Webserver started at http://127.1.221.130:44031/ using document root <none> and password file <none>
I20260810 22:28:43.429688 4091 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:43.429759 4091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:43.430011 4091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:43.432123 4091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "0b222cb3f76548e1b4d5d37a60b8e002"
format_stamp: "Formatted at 2026-08-10 22:28:43 on dist-test-slave-njxd"
server_key: "025ce521ce22c5309eae06ef2343585c"
server_key_iv: "d49688bc5147bb0e8dbf54cbbac9d1be"
server_key_version: "encryptionkey@0"
I20260810 22:28:43.432819 4091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "0b222cb3f76548e1b4d5d37a60b8e002"
format_stamp: "Formatted at 2026-08-10 22:28:43 on dist-test-slave-njxd"
server_key: "025ce521ce22c5309eae06ef2343585c"
server_key_iv: "d49688bc5147bb0e8dbf54cbbac9d1be"
server_key_version: "encryptionkey@0"
I20260810 22:28:43.436882 4091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:28:43.439550 4116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:43.440901 4091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:28:43.441046 4091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "0b222cb3f76548e1b4d5d37a60b8e002"
format_stamp: "Formatted at 2026-08-10 22:28:43 on dist-test-slave-njxd"
server_key: "025ce521ce22c5309eae06ef2343585c"
server_key_iv: "d49688bc5147bb0e8dbf54cbbac9d1be"
server_key_version: "encryptionkey@0"
I20260810 22:28:43.441171 4091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:43.455313 4091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:43.458751 4091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:43.458976 4091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:43.460546 4091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:43.460625 4091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:43.460700 4091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:43.460748 4091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:28:43 dist-test-slave-njxd krb5kdc[3806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:43.472299 4091 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:41699
I20260810 22:28:43.472373 4245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:41699 every 8 connection(s)
I20260810 22:28:43.473450 4091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:28:43.476125 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.461462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:41449 (local address 127.1.221.190:37029)
0810 22:28:43.461658 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:43.461664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:43.463199 (+ 1535us) server_negotiation.cc:407] Connection header received
0810 22:28:43.464465 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:43.464469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:43.464533 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:43.464618 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:43.466362 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.467297 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.469142 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.469543 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.469702 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.472559 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:43.472563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:43.472565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:43.472594 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:43.474184 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.474755 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.474758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.474760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.474853 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.475161 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.475164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.475165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.475350 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:43.475428 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:43.475899 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 22:28:43.476005 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 22:28:43.476989 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.462485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37029 (local address 127.1.221.130:41449)
0810 22:28:43.463023 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:43.463078 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:43.464234 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:43.464808 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:43.464818 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:43.465383 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:43.466155 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.466178 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.467464 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.467468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.469004 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.469015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.469741 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.469744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.469896 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.470536 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:43.470563 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:43.472313 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:43.474327 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.474333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.474348 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.474646 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.474938 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.474940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.474941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.475083 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.475442 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:43.475447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:43.475678 (+ 231us) client_negotiation.cc:769] Sending connection context
0810 22:28:43.475846 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 22:28:43.476104 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":179,"threads_started":1}
I20260810 22:28:43.478317 4247 heartbeater.cc:347] Connected to a master server at 127.1.221.190:37029
I20260810 22:28:43.478470 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 4091
I20260810 22:28:43.478498 4247 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:43.478595 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:28:43.478972 1910 external_mini_cluster.cc:1428] Setting key 2876cf0be408ef1ab4842cc509697276
I20260810 22:28:43.479208 4247 heartbeater.cc:511] Master 127.1.221.190:37029 requested a full tablet report, sending...
I20260810 22:28:43.480259 3857 ts_manager.cc:194] Registered new tserver with Master: 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130:41699)
I20260810 22:28:43.480937 3857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:41449
I20260810 22:28:43.489398 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.482142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:53097 (local address 127.1.221.190:37029)
0810 22:28:43.482287 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:43.482293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:43.482598 (+ 305us) server_negotiation.cc:407] Connection header received
0810 22:28:43.482702 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:43.482705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:43.482751 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:43.482846 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:43.483398 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.483886 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.484712 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.484951 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.485056 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.486215 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:43.486219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:43.486221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:43.486246 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:43.487697 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.488152 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.488156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.488158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.488242 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.488532 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.488535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.488537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.488750 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:43.488868 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:43.489130 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 22:28:43.489239 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:28:43.489420 4253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.482068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37029 (local address 127.1.221.130:53097)
0810 22:28:43.482489 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:43.482507 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:43.482620 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:43.482892 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:43.482898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:43.483166 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:43.483282 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.483288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.484020 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.484023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.484591 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.484599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.485071 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.485075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.485173 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.485714 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:43.485726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:43.486068 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:43.487833 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.487836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.487839 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.488046 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.488338 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.488342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.488344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.488431 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.488859 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:43.488862 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:43.488965 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 22:28:43.489076 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 22:28:43.489232 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:28:43.508975 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37029
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:43.622865 4258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:43.623150 4258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:43.623234 4258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:43.627424 4258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:43.627501 4258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:43.627575 4258 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 22:28:43.627617 4258 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:28:43.627693 4258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:28:43.632967 4258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:37029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:43.634168 4258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:43.635251 4258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:43.645082 4266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:43.645092 4268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:43.645092 4265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:43.645510 4258 server_base.cc:1034] running on GCE node
I20260810 22:28:43.646047 4258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:43.646642 4258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:43.647825 4258 hybrid_clock.cc:648] HybridClock initialized: now 1786400923647784 us; error 54 us; skew 500 ppm
Aug 10 22:28:43 dist-test-slave-njxd krb5kdc[3806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:43.650741 4258 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:28:43.651648 4258 webserver.cc:460] Webserver started at http://127.1.221.131:44057/ using document root <none> and password file <none>
I20260810 22:28:43.652194 4258 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:43.652254 4258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:43.652424 4258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:43.654543 4258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "ca5dce231a2c4852b2edad1aded895ad"
format_stamp: "Formatted at 2026-08-10 22:28:43 on dist-test-slave-njxd"
server_key: "85985811b87b28c03a6720c35fdc0b58"
server_key_iv: "3fa0fd9f664544304b15082f62bba7eb"
server_key_version: "encryptionkey@0"
I20260810 22:28:43.655177 4258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "ca5dce231a2c4852b2edad1aded895ad"
format_stamp: "Formatted at 2026-08-10 22:28:43 on dist-test-slave-njxd"
server_key: "85985811b87b28c03a6720c35fdc0b58"
server_key_iv: "3fa0fd9f664544304b15082f62bba7eb"
server_key_version: "encryptionkey@0"
I20260810 22:28:43.658996 4258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:28:43.661382 4275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:43.662758 4258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:28:43.662889 4258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "ca5dce231a2c4852b2edad1aded895ad"
format_stamp: "Formatted at 2026-08-10 22:28:43 on dist-test-slave-njxd"
server_key: "85985811b87b28c03a6720c35fdc0b58"
server_key_iv: "3fa0fd9f664544304b15082f62bba7eb"
server_key_version: "encryptionkey@0"
I20260810 22:28:43.663002 4258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:43.678447 4258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:43.681306 4258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:43.681514 4258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:43.683090 4258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:43.683166 4258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:43.683241 4258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:43.683291 4258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:28:43 dist-test-slave-njxd krb5kdc[3806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400923, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:43.694653 4258 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:32827
I20260810 22:28:43.694716 4402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:32827 every 8 connection(s)
I20260810 22:28:43.695851 4258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:28:43.696559 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 4258
I20260810 22:28:43.696676 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:28:43.697003 1910 external_mini_cluster.cc:1428] Setting key afb2723b925102ea104d0ae975f62172
I20260810 22:28:43.698061 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.684013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:38817 (local address 127.1.221.190:37029)
0810 22:28:43.684185 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:43.684190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:43.685092 (+ 902us) server_negotiation.cc:407] Connection header received
0810 22:28:43.686384 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:43.686389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:43.686454 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:43.686518 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:43.688039 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.688604 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.690071 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.690332 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.690458 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.693453 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:43.693457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:43.693459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:43.693497 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:43.695251 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.695771 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.695775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.695777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.695884 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.696484 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.696489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.696490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.696748 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:43.696861 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:43.697725 (+ 864us) server_negotiation.cc:299] Negotiation successful
0810 22:28:43.697895 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:28:43.698801 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.684465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37029 (local address 127.1.221.131:38817)
0810 22:28:43.684943 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:43.684994 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:43.686121 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:43.686889 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:43.686895 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:43.687211 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:43.687872 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.687897 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.688742 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.688746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.689925 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.689935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.690513 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.690517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.690709 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.691347 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:43.691373 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:43.693279 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:43.695383 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.695387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.695399 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.695655 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.696095 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.696098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.696102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.696354 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.696867 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:43.696873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:43.697107 (+ 234us) client_negotiation.cc:769] Sending connection context
0810 22:28:43.697327 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 22:28:43.697719 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":124,"threads_started":1}
I20260810 22:28:43.700096 4403 heartbeater.cc:347] Connected to a master server at 127.1.221.190:37029
I20260810 22:28:43.700260 4403 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:43.700826 4403 heartbeater.cc:511] Master 127.1.221.190:37029 requested a full tablet report, sending...
I20260810 22:28:43.701826 3856 ts_manager.cc:194] Registered new tserver with Master: ca5dce231a2c4852b2edad1aded895ad (127.1.221.131:32827)
I20260810 22:28:43.702541 3856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:38817
I20260810 22:28:43.708997 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.702252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:36075 (local address 127.1.221.190:37029)
0810 22:28:43.702369 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:43.702379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:43.702592 (+ 213us) server_negotiation.cc:407] Connection header received
0810 22:28:43.702724 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:43.702727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:43.702770 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:43.702871 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:43.703321 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.703850 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.704704 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.704920 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.705025 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.706128 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:43.706131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:43.706143 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:43.706173 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:43.707370 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.707788 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.707793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.707795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.707874 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.708154 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.708157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.708158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.708351 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:43.708459 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:43.708730 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 22:28:43.708839 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 22:28:43.709009 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.702181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37029 (local address 127.1.221.131:36075)
0810 22:28:43.702504 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:43.702516 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:43.702595 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:43.702890 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:43.702893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:43.703093 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:43.703191 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.703196 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.703979 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.703983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.704551 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.704559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.705040 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.705043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.705141 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.705664 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:43.705677 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:43.705979 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:43.707481 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.707485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.707487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.707690 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.707966 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.707970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.707972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.708056 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.708449 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:43.708452 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:43.708554 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 22:28:43.708674 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 22:28:43.708833 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":95,"threads_started":1}
I20260810 22:28:43.711659 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 22:28:43 dist-test-slave-njxd krb5kdc[3806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:28:43 dist-test-slave-njxd krb5kdc[3806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:43.739449 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.731076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.1.221.190:37029)
0810 22:28:43.731228 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:43.731231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:43.731244 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:28:43.731287 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:43.731290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:43.731331 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:43.731430 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:43.732199 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.732718 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.733735 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.733947 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.734058 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.736486 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:43.736489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:43.736491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:43.736515 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:43.737714 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.738133 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.738136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.738138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.738224 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.738537 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.738540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.738542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.738744 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:43.738890 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:43.739149 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 22:28:43.739276 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 22:28:43.742273 3857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:28:43.745612 3857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:28:43.760425 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.754004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36241 (local address 127.0.0.1:55990)
0810 22:28:43.754388 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:43.754446 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:43.754664 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:43.755721 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:43.755724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:43.755757 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:43.755939 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.755951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.757169 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.757174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.758647 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.758659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.759975 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.759979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.760073 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.760097 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:28:43.760189 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 22:28:43.760265 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":61,"spinlock_wait_cycles":5504,"thread_start_us":147,"threads_started":1}
I20260810 22:28:43.760656 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.754468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.1.221.130:41699)
0810 22:28:43.754949 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:43.754962 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:43.754990 (+ 28us) server_negotiation.cc:407] Connection header received
0810 22:28:43.755068 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:43.755074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:43.755302 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:43.755474 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:43.756251 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.757380 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.758817 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.759513 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.759628 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.759802 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:43.760357 (+ 555us) server_negotiation.cc:299] Negotiation successful
0810 22:28:43.760474 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":157,"threads_started":1}
I20260810 22:28:43.760488 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.754837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.1.221.129:36241)
0810 22:28:43.755206 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:43.755221 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:43.755243 (+ 22us) server_negotiation.cc:407] Connection header received
0810 22:28:43.755343 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:43.755349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:43.755542 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:43.755938 (+ 396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:43.756085 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.757038 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.758955 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.759843 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.759961 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.760101 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:43.760235 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 22:28:43.760354 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":97,"threads_started":1}
I20260810 22:28:43.760262 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.753958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:41699 (local address 127.0.0.1:60510)
0810 22:28:43.754387 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:43.754435 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:43.754592 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:43.755711 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:43.755719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:43.755748 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:43.756091 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.756099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.757499 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.757503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.758667 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.758673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.759643 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.759646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.759773 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.759822 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 22:28:43.759935 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 22:28:43.760069 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":2,"spinlock_wait_cycles":7808,"thread_start_us":81,"threads_started":1}
I20260810 22:28:43.761251 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.755343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:32827 (local address 127.0.0.1:38368)
0810 22:28:43.756198 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:43.756215 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:43.756317 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:43.756975 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:43.756978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:43.756995 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:43.757073 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.757078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.758260 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.758264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.759390 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.759401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.760702 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.760708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.760827 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.760852 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 22:28:43.761008 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 22:28:43.761082 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":96,"threads_started":1}
I20260810 22:28:43.761538 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.755520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.1.221.131:32827)
0810 22:28:43.756026 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:43.756038 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:43.756421 (+ 383us) server_negotiation.cc:407] Connection header received
0810 22:28:43.756505 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:43.756510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:43.756691 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:43.756836 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:43.757195 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.758119 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.759532 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.760373 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.760476 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.760627 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:43.761285 (+ 658us) server_negotiation.cc:299] Negotiation successful
0810 22:28:43.761382 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":166,"threads_started":1}
I20260810 22:28:43.763973 4006 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee99a13e414454492151ac4053b12f2 (DEFAULT_TABLE table=test-table [id=cf0933565ab340a4b344f1bc70130bae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:43.764027 4173 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee99a13e414454492151ac4053b12f2 (DEFAULT_TABLE table=test-table [id=cf0933565ab340a4b344f1bc70130bae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:43.765094 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee99a13e414454492151ac4053b12f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:43.765128 4173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee99a13e414454492151ac4053b12f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:43.764979 4331 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee99a13e414454492151ac4053b12f2 (DEFAULT_TABLE table=test-table [id=cf0933565ab340a4b344f1bc70130bae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:43.765982 4331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee99a13e414454492151ac4053b12f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:43.771692 4428 tablet_bootstrap.cc:492] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8: Bootstrap starting.
I20260810 22:28:43.773294 4429 tablet_bootstrap.cc:492] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002: Bootstrap starting.
I20260810 22:28:43.774194 4428 tablet_bootstrap.cc:654] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:43.774351 4430 tablet_bootstrap.cc:492] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad: Bootstrap starting.
I20260810 22:28:43.775117 4428 log.cc:824] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:43.775665 4429 tablet_bootstrap.cc:654] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:43.776353 4430 tablet_bootstrap.cc:654] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:43.776583 4429 log.cc:824] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:43.777292 4430 log.cc:824] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:43.777729 4428 tablet_bootstrap.cc:492] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8: No bootstrap required, opened a new log
I20260810 22:28:43.777956 4428 ts_tablet_manager.cc:1397] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:28:43.778741 4429 tablet_bootstrap.cc:492] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002: No bootstrap required, opened a new log
I20260810 22:28:43.778937 4429 ts_tablet_manager.cc:1397] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:28:43.779340 4430 tablet_bootstrap.cc:492] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad: No bootstrap required, opened a new log
I20260810 22:28:43.779517 4430 ts_tablet_manager.cc:1397] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 22:28:43.781826 4428 raft_consensus.cc:348] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.782065 4428 raft_consensus.cc:374] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:43.782137 4428 raft_consensus.cc:729] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880c64863b3a4e5d850578fb56f35bd8, State: Initialized, Role: FOLLOWER
I20260810 22:28:43.782599 4428 consensus_queue.cc:260] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.783313 4081 heartbeater.cc:503] Master 127.1.221.190:37029 was elected leader, sending a full tablet report...
I20260810 22:28:43.783293 4430 raft_consensus.cc:348] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.783538 4430 raft_consensus.cc:374] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:43.783592 4430 raft_consensus.cc:729] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5dce231a2c4852b2edad1aded895ad, State: Initialized, Role: FOLLOWER
I20260810 22:28:43.782918 4429 raft_consensus.cc:348] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.783701 4429 raft_consensus.cc:374] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:43.783761 4429 raft_consensus.cc:729] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b222cb3f76548e1b4d5d37a60b8e002, State: Initialized, Role: FOLLOWER
I20260810 22:28:43.784049 4430 consensus_queue.cc:260] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.783615 4428 ts_tablet_manager.cc:1428] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:28:43.784161 4429 consensus_queue.cc:260] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.785004 4403 heartbeater.cc:503] Master 127.1.221.190:37029 was elected leader, sending a full tablet report...
I20260810 22:28:43.785130 4429 ts_tablet_manager.cc:1428] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 22:28:43.785120 4430 ts_tablet_manager.cc:1428] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:28:43.786398 4247 heartbeater.cc:503] Master 127.1.221.190:37029 was elected leader, sending a full tablet report...
I20260810 22:28:43.828471 4437 raft_consensus.cc:482] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:28:43.828730 4437 raft_consensus.cc:504] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.830227 4437 leader_election.cc:288] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 880c64863b3a4e5d850578fb56f35bd8 (127.1.221.129:36241), ca5dce231a2c4852b2edad1aded895ad (127.1.221.131:32827)
I20260810 22:28:43.834786 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.830537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36241 (local address 127.1.221.130:35265)
0810 22:28:43.830760 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:43.830777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:43.830894 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:43.831174 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:43.831177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:43.831197 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:43.831366 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.831373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.832366 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.832368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.833750 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.833763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.834427 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.834430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.834510 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.834530 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 22:28:43.834619 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 22:28:43.834677 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":1}
I20260810 22:28:43.834875 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.830656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:35265 (local address 127.1.221.129:36241)
0810 22:28:43.830785 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:43.830789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:43.830947 (+ 158us) server_negotiation.cc:407] Connection header received
0810 22:28:43.831010 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:43.831012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:43.831059 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:43.831135 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:43.831474 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.832256 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.833876 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.834310 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.834435 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.834488 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:43.834657 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 22:28:43.834737 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:28:43.835574 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee99a13e414454492151ac4053b12f2" candidate_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "880c64863b3a4e5d850578fb56f35bd8" is_pre_election: true
I20260810 22:28:43.835752 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.830537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:32827 (local address 127.1.221.130:56915)
0810 22:28:43.831204 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:43.831218 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:43.831310 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:43.831551 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:43.831553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:43.831569 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:43.831669 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.831674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.832553 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.832556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.834204 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.834213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.835328 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.835340 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.835418 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.835437 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:28:43.835534 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 22:28:43.835603 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":275,"thread_start_us":158,"threads_started":1}
I20260810 22:28:43.835742 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.830758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:56915 (local address 127.1.221.131:32827)
0810 22:28:43.830907 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:43.830911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:43.831311 (+ 400us) server_negotiation.cc:407] Connection header received
0810 22:28:43.831398 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:43.831401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:43.831451 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:43.831529 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:43.831782 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.832459 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.834333 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.834959 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.835081 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.835120 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:43.835556 (+ 436us) server_negotiation.cc:299] Negotiation successful
0810 22:28:43.835636 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:28:43.835933 4028 raft_consensus.cc:2454] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b222cb3f76548e1b4d5d37a60b8e002 in term 0.
I20260810 22:28:43.836318 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee99a13e414454492151ac4053b12f2" candidate_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5dce231a2c4852b2edad1aded895ad" is_pre_election: true
I20260810 22:28:43.836452 4121 leader_election.cc:302] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b222cb3f76548e1b4d5d37a60b8e002, 880c64863b3a4e5d850578fb56f35bd8; no voters:
I20260810 22:28:43.836627 4352 raft_consensus.cc:2454] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b222cb3f76548e1b4d5d37a60b8e002 in term 0.
I20260810 22:28:43.836709 4437 raft_consensus.cc:2781] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:28:43.836802 4437 raft_consensus.cc:482] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:28:43.836856 4437 raft_consensus.cc:3037] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:43.838091 4437 raft_consensus.cc:504] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.838430 4437 leader_election.cc:288] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [CANDIDATE]: Term 1 election: Requested vote from peers 880c64863b3a4e5d850578fb56f35bd8 (127.1.221.129:36241), ca5dce231a2c4852b2edad1aded895ad (127.1.221.131:32827)
I20260810 22:28:43.838831 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee99a13e414454492151ac4053b12f2" candidate_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "880c64863b3a4e5d850578fb56f35bd8"
I20260810 22:28:43.838824 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee99a13e414454492151ac4053b12f2" candidate_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5dce231a2c4852b2edad1aded895ad"
I20260810 22:28:43.838955 4352 raft_consensus.cc:3037] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:43.838955 4028 raft_consensus.cc:3037] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:43.840293 4352 raft_consensus.cc:2454] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b222cb3f76548e1b4d5d37a60b8e002 in term 1.
I20260810 22:28:43.840299 4028 raft_consensus.cc:2454] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b222cb3f76548e1b4d5d37a60b8e002 in term 1.
I20260810 22:28:43.840641 4121 leader_election.cc:302] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b222cb3f76548e1b4d5d37a60b8e002, 880c64863b3a4e5d850578fb56f35bd8, ca5dce231a2c4852b2edad1aded895ad; no voters:
I20260810 22:28:43.840790 4437 raft_consensus.cc:2781] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:43.841046 4437 raft_consensus.cc:686] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 LEADER]: Becoming Leader. State: Replica: 0b222cb3f76548e1b4d5d37a60b8e002, State: Running, Role: LEADER
I20260810 22:28:43.841414 4437 consensus_queue.cc:237] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.844511 3855 catalog_manager.cc:5495] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130). New cstate: current_term: 1 leader_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } health_report { overall_health: UNKNOWN } } }
Aug 10 22:28:43 dist-test-slave-njxd krb5kdc[3806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400922, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:43.883903 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.875550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45962 (local address 127.1.221.190:37029)
0810 22:28:43.875709 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:43.875714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:43.875859 (+ 145us) server_negotiation.cc:407] Connection header received
0810 22:28:43.876022 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:43.876026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:43.876070 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:43.876157 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:43.876853 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.877369 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.878261 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.878475 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.878582 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.880853 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:43.880864 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:43.880866 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:43.880887 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:43.882118 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.882587 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.882591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.882593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.882678 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:43.882948 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:43.882953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:43.882955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:43.883137 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:43.883247 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:43.883538 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 22:28:43.883756 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":49}
I20260810 22:28:43.883903 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:43.875458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37029 (local address 127.0.0.1:45962)
0810 22:28:43.875758 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:43.875775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:43.875914 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:43.876178 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:43.876180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:43.876630 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:43.876727 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.876734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.877478 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.877484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.878135 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:43.878144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:43.878586 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:43.878589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:43.878667 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:43.879246 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:43.879259 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:43.880714 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:43.882229 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.882242 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.882249 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.882485 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.882769 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:43.882774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:43.882776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:43.882859 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:43.883247 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:43.883256 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:43.883374 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 22:28:43.883493 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 22:28:43.883691 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":4,"thread_start_us":98,"threads_started":1}
I20260810 22:28:43.888149 3855 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:45962:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 22:28:43.888967 3855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:28:43.893939 4006 tablet_service.cc:1467] Processing CreateTablet for tablet 4f07dad36ae241e495ac6a0b851e7fcd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e9f4067ce22436c9457c2b7ab8fa0a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:28:43.894061 4173 tablet_service.cc:1467] Processing CreateTablet for tablet 4f07dad36ae241e495ac6a0b851e7fcd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e9f4067ce22436c9457c2b7ab8fa0a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:28:43.894749 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f07dad36ae241e495ac6a0b851e7fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:43.894379 4331 tablet_service.cc:1467] Processing CreateTablet for tablet 4f07dad36ae241e495ac6a0b851e7fcd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e9f4067ce22436c9457c2b7ab8fa0a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:28:43.894881 4173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f07dad36ae241e495ac6a0b851e7fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:43.895012 4331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f07dad36ae241e495ac6a0b851e7fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:43.897444 4428 tablet_bootstrap.cc:492] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8: Bootstrap starting.
I20260810 22:28:43.897969 4430 tablet_bootstrap.cc:492] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad: Bootstrap starting.
I20260810 22:28:43.897969 4429 tablet_bootstrap.cc:492] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002: Bootstrap starting.
I20260810 22:28:43.898605 4428 tablet_bootstrap.cc:654] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:43.898933 4429 tablet_bootstrap.cc:654] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:43.898933 4430 tablet_bootstrap.cc:654] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:43.900188 4428 tablet_bootstrap.cc:492] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8: No bootstrap required, opened a new log
I20260810 22:28:43.900286 4428 ts_tablet_manager.cc:1397] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:28:43.900362 4429 tablet_bootstrap.cc:492] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002: No bootstrap required, opened a new log
I20260810 22:28:43.900442 4429 ts_tablet_manager.cc:1397] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 22:28:43.900547 4430 tablet_bootstrap.cc:492] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad: No bootstrap required, opened a new log
I20260810 22:28:43.900643 4430 ts_tablet_manager.cc:1397] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:28:43.900799 4428 raft_consensus.cc:348] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.900923 4428 raft_consensus.cc:374] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:43.900957 4428 raft_consensus.cc:729] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880c64863b3a4e5d850578fb56f35bd8, State: Initialized, Role: FOLLOWER
I20260810 22:28:43.900898 4429 raft_consensus.cc:348] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.901006 4429 raft_consensus.cc:374] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:43.901041 4429 raft_consensus.cc:729] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b222cb3f76548e1b4d5d37a60b8e002, State: Initialized, Role: FOLLOWER
I20260810 22:28:43.901100 4428 consensus_queue.cc:260] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.901118 4430 raft_consensus.cc:348] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.901157 4429 consensus_queue.cc:260] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.901222 4430 raft_consensus.cc:374] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:43.901269 4430 raft_consensus.cc:729] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5dce231a2c4852b2edad1aded895ad, State: Initialized, Role: FOLLOWER
I20260810 22:28:43.901415 4428 ts_tablet_manager.cc:1428] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:28:43.901388 4430 consensus_queue.cc:260] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:43.901453 4437 tablet_replica.cc:439] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } } }
I20260810 22:28:43.901441 4435 tablet_replica.cc:439] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } } }
I20260810 22:28:43.901628 4437 tablet_replica.cc:442] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:43.901624 4435 tablet_replica.cc:442] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:43.901861 4438 tablet_replica.cc:439] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } } }
I20260810 22:28:43.901999 4438 tablet_replica.cc:442] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:43.902275 4430 ts_tablet_manager.cc:1428] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 22:28:43.902508 4429 ts_tablet_manager.cc:1428] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260810 22:28:43.946969 4404 tablet.cc:2369] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:28:43.974944 4248 tablet.cc:2369] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:28:43.993420 4082 tablet.cc:2369] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:28:44.207839 4435 raft_consensus.cc:482] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:28:44.208006 4435 raft_consensus.cc:504] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:44.209295 4435 leader_election.cc:288] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130:41699), ca5dce231a2c4852b2edad1aded895ad (127.1.221.131:32827)
I20260810 22:28:44.215072 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.210270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:34201 (local address 127.1.221.131:32827)
0810 22:28:44.210399 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:44.210404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:44.210625 (+ 221us) server_negotiation.cc:407] Connection header received
0810 22:28:44.210796 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:44.210799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:44.210846 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:44.210925 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:44.211204 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.211869 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.213734 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.214349 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.214478 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.214532 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:44.214863 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 22:28:44.214936 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 22:28:44.215072 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.209727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:53539 (local address 127.1.221.130:41699)
0810 22:28:44.209905 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:44.209911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:44.210064 (+ 153us) server_negotiation.cc:407] Connection header received
0810 22:28:44.210326 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:44.210330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:44.210389 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:44.210477 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:44.210797 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.211583 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.213734 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.214349 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.214478 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.214532 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:44.214863 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 22:28:44.214940 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 22:28:44.215246 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.209558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:41699 (local address 127.1.221.129:53539)
0810 22:28:44.209966 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:44.209979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:44.210163 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:44.210499 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:44.210503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:44.210522 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:44.210680 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.210688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.211684 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.211687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.213604 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.213618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.214478 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.214481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.214703 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.214750 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 22:28:44.214831 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 22:28:44.214961 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":171,"thread_start_us":173,"threads_started":1}
I20260810 22:28:44.215682 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.210152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:32827 (local address 127.1.221.129:34201)
0810 22:28:44.210530 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:44.210543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:44.210663 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:44.210957 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:44.210959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:44.210977 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:44.211091 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.211096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.211978 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.211983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.213604 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.213619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.214478 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.214481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.214700 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.214752 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 22:28:44.214831 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 22:28:44.214961 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":242,"thread_start_us":119,"threads_started":1}
I20260810 22:28:44.215870 4196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f07dad36ae241e495ac6a0b851e7fcd" candidate_uuid: "880c64863b3a4e5d850578fb56f35bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" is_pre_election: true
I20260810 22:28:44.216104 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f07dad36ae241e495ac6a0b851e7fcd" candidate_uuid: "880c64863b3a4e5d850578fb56f35bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5dce231a2c4852b2edad1aded895ad" is_pre_election: true
I20260810 22:28:44.216164 4196 raft_consensus.cc:2454] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 880c64863b3a4e5d850578fb56f35bd8 in term 0.
I20260810 22:28:44.216266 4352 raft_consensus.cc:2454] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 880c64863b3a4e5d850578fb56f35bd8 in term 0.
I20260810 22:28:44.216712 3954 leader_election.cc:302] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b222cb3f76548e1b4d5d37a60b8e002, 880c64863b3a4e5d850578fb56f35bd8; no voters:
I20260810 22:28:44.217010 4435 raft_consensus.cc:2781] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:28:44.217072 4435 raft_consensus.cc:482] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:28:44.217101 4435 raft_consensus.cc:3037] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:44.218328 4435 raft_consensus.cc:504] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:44.218691 4435 leader_election.cc:288] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130:41699), ca5dce231a2c4852b2edad1aded895ad (127.1.221.131:32827)
I20260810 22:28:44.219128 4196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f07dad36ae241e495ac6a0b851e7fcd" candidate_uuid: "880c64863b3a4e5d850578fb56f35bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b222cb3f76548e1b4d5d37a60b8e002"
I20260810 22:28:44.219180 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f07dad36ae241e495ac6a0b851e7fcd" candidate_uuid: "880c64863b3a4e5d850578fb56f35bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5dce231a2c4852b2edad1aded895ad"
I20260810 22:28:44.219234 4196 raft_consensus.cc:3037] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:44.219326 4352 raft_consensus.cc:3037] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:44.220284 4352 raft_consensus.cc:2454] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 880c64863b3a4e5d850578fb56f35bd8 in term 1.
I20260810 22:28:44.220373 4196 raft_consensus.cc:2454] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 880c64863b3a4e5d850578fb56f35bd8 in term 1.
I20260810 22:28:44.220615 3954 leader_election.cc:302] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 880c64863b3a4e5d850578fb56f35bd8, ca5dce231a2c4852b2edad1aded895ad; no voters:
I20260810 22:28:44.220786 4435 raft_consensus.cc:2781] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:44.221010 4435 raft_consensus.cc:686] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [term 1 LEADER]: Becoming Leader. State: Replica: 880c64863b3a4e5d850578fb56f35bd8, State: Running, Role: LEADER
I20260810 22:28:44.221388 4435 consensus_queue.cc:237] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:44.223793 4456 tablet_replica.cc:439] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8: TxnStatusTablet state changed. Reason: New leader 880c64863b3a4e5d850578fb56f35bd8. Latest consensus state: current_term: 1 leader_uuid: "880c64863b3a4e5d850578fb56f35bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } } }
I20260810 22:28:44.223912 4456 tablet_replica.cc:442] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:28:44.224223 4458 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 22:28:44.224442 4458 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 22:28:44.224944 3855 catalog_manager.cc:5495] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 880c64863b3a4e5d850578fb56f35bd8 (127.1.221.129). New cstate: current_term: 1 leader_uuid: "880c64863b3a4e5d850578fb56f35bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } health_report { overall_health: UNKNOWN } } }
I20260810 22:28:44.269078 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.264982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36241 (local address 127.0.0.1:55994)
0810 22:28:44.265190 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:44.265208 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:44.265316 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:44.265754 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:44.265757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:44.265765 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:28:44.265874 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.265879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.266646 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.266651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.267426 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.267435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.267901 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.267905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.268001 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.268055 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:28:44.268771 (+ 716us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:28:44.268790 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:28:44.268863 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 22:28:44.268950 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 22:28:44.269078 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.265149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.1.221.129:36241)
0810 22:28:44.265294 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:44.265299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:44.265317 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:28:44.265484 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:44.265487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:44.265543 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:44.265728 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:28:44.265982 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.266526 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.267553 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.267784 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.267897 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.268221 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:28:44.268662 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:28:44.268796 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:44.268873 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 22:28:44.268927 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 22:28:44.275126 4196 raft_consensus.cc:1261] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Refusing update from remote peer 880c64863b3a4e5d850578fb56f35bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:44.275138 4352 raft_consensus.cc:1261] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Refusing update from remote peer 880c64863b3a4e5d850578fb56f35bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:44.275816 4456 consensus_queue.cc:1035] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:44.276144 4435 consensus_queue.cc:1035] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:44.279844 4437 tablet_replica.cc:439] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002: TxnStatusTablet state changed. Reason: New leader 880c64863b3a4e5d850578fb56f35bd8. Latest consensus state: current_term: 1 leader_uuid: "880c64863b3a4e5d850578fb56f35bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } } }
I20260810 22:28:44.279968 4437 tablet_replica.cc:442] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:44.280601 4438 tablet_replica.cc:439] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad: TxnStatusTablet state changed. Reason: New leader 880c64863b3a4e5d850578fb56f35bd8. Latest consensus state: current_term: 1 leader_uuid: "880c64863b3a4e5d850578fb56f35bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } } }
I20260810 22:28:44.280714 4438 tablet_replica.cc:442] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:44.281271 4456 tablet_replica.cc:439] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "880c64863b3a4e5d850578fb56f35bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } } }
I20260810 22:28:44.281410 4456 tablet_replica.cc:442] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:28:44.282043 4461 mvcc.cc:205] Tried to move back new op lower bound from 7317098185823236096 to 7317098185613963264. Current Snapshot: MvccSnapshot[applied={T|T < 7317098185823236096}]
I20260810 22:28:44.282497 4437 tablet_replica.cc:439] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "880c64863b3a4e5d850578fb56f35bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } } }
I20260810 22:28:44.282604 4437 tablet_replica.cc:442] T 4f07dad36ae241e495ac6a0b851e7fcd P 0b222cb3f76548e1b4d5d37a60b8e002: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:44.286616 4462 mvcc.cc:205] Tried to move back new op lower bound from 7317098185823236096 to 7317098185613963264. Current Snapshot: MvccSnapshot[applied={T|T < 7317098185823236096}]
I20260810 22:28:44.288726 4435 tablet_replica.cc:439] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "880c64863b3a4e5d850578fb56f35bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } } }
I20260810 22:28:44.288848 4435 tablet_replica.cc:442] T 4f07dad36ae241e495ac6a0b851e7fcd P 880c64863b3a4e5d850578fb56f35bd8: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:28:44.290097 4438 tablet_replica.cc:439] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "880c64863b3a4e5d850578fb56f35bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } } }
I20260810 22:28:44.290221 4438 tablet_replica.cc:442] T 4f07dad36ae241e495ac6a0b851e7fcd P ca5dce231a2c4852b2edad1aded895ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:28:44.290879 4463 mvcc.cc:205] Tried to move back new op lower bound from 7317098185823236096 to 7317098185613963264. Current Snapshot: MvccSnapshot[applied={T|T < 7317098185823236096}]
I20260810 22:28:44.294133 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.289080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.1.221.130:41699)
0810 22:28:44.289186 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:44.289191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:44.289919 (+ 728us) server_negotiation.cc:407] Connection header received
0810 22:28:44.290181 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:44.290184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:44.290258 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:44.290339 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:28:44.290811 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.291389 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.292410 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.292742 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.292848 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.293244 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:28:44.293783 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:28:44.293902 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:44.293942 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 22:28:44.294000 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 22:28:44.305373 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.301725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:37567 (local address 127.1.221.129:36241)
0810 22:28:44.301872 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:44.301877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:44.302134 (+ 257us) server_negotiation.cc:407] Connection header received
0810 22:28:44.302263 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:44.302266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:44.302311 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:44.302366 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:28:44.302649 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.303163 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.304298 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.304526 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.304602 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.304969 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:28:44.305038 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:28:44.305091 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:44.305217 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 22:28:44.305265 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:28:44.305462 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.301668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36241 (local address 127.1.221.130:37567)
0810 22:28:44.302028 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:44.302044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:44.302153 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:44.302400 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:44.302403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:44.302410 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:28:44.302536 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.302542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.303291 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.303295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.304167 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.304178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.304650 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.304654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.304752 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.304808 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:28:44.305123 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:28:44.305130 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 22:28:44.305199 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 22:28:44.305291 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":123,"threads_started":1}
I20260810 22:28:44.312585 4352 raft_consensus.cc:1261] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Refusing update from remote peer 0b222cb3f76548e1b4d5d37a60b8e002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:44.312814 4028 raft_consensus.cc:1261] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Refusing update from remote peer 0b222cb3f76548e1b4d5d37a60b8e002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:44.313112 4440 consensus_queue.cc:1035] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:44.313378 4437 consensus_queue.cc:1035] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:44.340013 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.336059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:34679 (local address 127.1.221.130:41699)
0810 22:28:44.336173 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:44.336178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:44.336446 (+ 268us) server_negotiation.cc:407] Connection header received
0810 22:28:44.336529 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:44.336533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:44.336587 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:44.336675 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:28:44.336867 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.337701 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.338969 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.339193 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.339269 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.339616 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:28:44.339684 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:28:44.339738 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:44.339861 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 22:28:44.339907 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:28:44.340117 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.335946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:41699 (local address 127.1.221.129:34679)
0810 22:28:44.336356 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:44.336368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:44.336457 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:44.336676 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:44.336678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:44.336683 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:28:44.336772 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.336775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.337842 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.337845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.338835 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.338846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.339314 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.339318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.339419 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.339476 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:28:44.339767 (+ 291us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:28:44.339773 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:28:44.339844 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 22:28:44.339936 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":151,"threads_started":1}
I20260810 22:28:44.369961 3857 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:28:44.370175 3857 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:28:44.373222 3857 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d: Sending DeleteTablet for 3 replicas of tablet 3ee99a13e414454492151ac4053b12f2
I20260810 22:28:44.374047 4173 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee99a13e414454492151ac4053b12f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:44 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:60510
I20260810 22:28:44.374738 4484 tablet_replica.cc:330] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002: stopping tablet replica
I20260810 22:28:44.374819 4006 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee99a13e414454492151ac4053b12f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:44 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:55990
I20260810 22:28:44.375218 4484 raft_consensus.cc:2229] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:28:44.375663 4484 raft_consensus.cc:2258] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:44.375720 4331 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee99a13e414454492151ac4053b12f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:44 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:38368
I20260810 22:28:44.377007 3857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:28:44.377377 4484 ts_tablet_manager.cc:1905] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 22:28:44.377465 3857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:28:44.378988 4487 tablet_replica.cc:330] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad: stopping tablet replica
I20260810 22:28:44.379333 4487 raft_consensus.cc:2229] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:28:44.379567 4487 raft_consensus.cc:2258] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:44.380971 4487 ts_tablet_manager.cc:1905] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:44.383342 4173 tablet_service.cc:1467] Processing CreateTablet for tablet 3860edc1a1984509993646611e8cdef1 (DEFAULT_TABLE table=test-table [id=7338312f9d204019abd3257356dc8d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:44.383708 4173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3860edc1a1984509993646611e8cdef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:44.385255 4484 ts_tablet_manager.cc:1918] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:28:44.385430 4484 log.cc:1196] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/3ee99a13e414454492151ac4053b12f2
I20260810 22:28:44.385866 4484 ts_tablet_manager.cc:1939] T 3ee99a13e414454492151ac4053b12f2 P 0b222cb3f76548e1b4d5d37a60b8e002: Deleting consensus metadata
I20260810 22:28:44.386133 4331 tablet_service.cc:1467] Processing CreateTablet for tablet 3860edc1a1984509993646611e8cdef1 (DEFAULT_TABLE table=test-table [id=7338312f9d204019abd3257356dc8d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:44.386397 4331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3860edc1a1984509993646611e8cdef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:44.388055 4429 tablet_bootstrap.cc:492] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002: Bootstrap starting.
I20260810 22:28:44.388082 3838 catalog_manager.cc:4841] TS 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130:41699): tablet 3ee99a13e414454492151ac4053b12f2 (table test-table [id=cf0933565ab340a4b344f1bc70130bae]) successfully deleted
I20260810 22:28:44.388783 4430 tablet_bootstrap.cc:492] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad: Bootstrap starting.
I20260810 22:28:44.389067 4429 tablet_bootstrap.cc:654] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:44.389707 4430 tablet_bootstrap.cc:654] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:44.390501 4429 tablet_bootstrap.cc:492] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002: No bootstrap required, opened a new log
I20260810 22:28:44.390614 4429 ts_tablet_manager.cc:1397] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 22:28:44.390975 4430 tablet_bootstrap.cc:492] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad: No bootstrap required, opened a new log
I20260810 22:28:44.391067 4430 ts_tablet_manager.cc:1397] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:28:44.391144 4429 raft_consensus.cc:348] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:44.391270 4429 raft_consensus.cc:374] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:44.391310 4429 raft_consensus.cc:729] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b222cb3f76548e1b4d5d37a60b8e002, State: Initialized, Role: FOLLOWER
I20260810 22:28:44.391443 4429 consensus_queue.cc:260] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:44.391563 4430 raft_consensus.cc:348] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:44.391681 4430 raft_consensus.cc:374] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:44.391728 4430 raft_consensus.cc:729] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5dce231a2c4852b2edad1aded895ad, State: Initialized, Role: FOLLOWER
I20260810 22:28:44.391857 4430 consensus_queue.cc:260] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:44.392128 4430 ts_tablet_manager.cc:1428] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:28:44.392709 4429 ts_tablet_manager.cc:1428] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 22:28:44.393132 4006 tablet_service.cc:1467] Processing CreateTablet for tablet 3860edc1a1984509993646611e8cdef1 (DEFAULT_TABLE table=test-table [id=7338312f9d204019abd3257356dc8d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:44.393482 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3860edc1a1984509993646611e8cdef1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:44.396351 4428 tablet_bootstrap.cc:492] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8: Bootstrap starting.
I20260810 22:28:44.397373 4428 tablet_bootstrap.cc:654] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:44.397647 4486 tablet_replica.cc:330] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8: stopping tablet replica
I20260810 22:28:44.398025 4486 raft_consensus.cc:2229] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:28:44.398195 4486 raft_consensus.cc:2258] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:44.399541 4428 tablet_bootstrap.cc:492] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8: No bootstrap required, opened a new log
I20260810 22:28:44.399645 4428 ts_tablet_manager.cc:1397] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 22:28:44.400111 4428 raft_consensus.cc:348] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:44.400230 4428 raft_consensus.cc:374] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:44.400264 4428 raft_consensus.cc:729] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880c64863b3a4e5d850578fb56f35bd8, State: Initialized, Role: FOLLOWER
I20260810 22:28:44.400410 4428 consensus_queue.cc:260] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:44.400702 4428 ts_tablet_manager.cc:1428] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:28:44.401412 4486 ts_tablet_manager.cc:1905] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:44.404719 4486 ts_tablet_manager.cc:1918] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:28:44.404830 4486 log.cc:1196] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/wal/wals/3ee99a13e414454492151ac4053b12f2
I20260810 22:28:44.405145 4486 ts_tablet_manager.cc:1939] T 3ee99a13e414454492151ac4053b12f2 P 880c64863b3a4e5d850578fb56f35bd8: Deleting consensus metadata
I20260810 22:28:44.406445 3840 catalog_manager.cc:4841] TS 880c64863b3a4e5d850578fb56f35bd8 (127.1.221.129:36241): tablet 3ee99a13e414454492151ac4053b12f2 (table test-table [id=cf0933565ab340a4b344f1bc70130bae]) successfully deleted
I20260810 22:28:44.408028 4487 ts_tablet_manager.cc:1918] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:28:44.408149 4487 log.cc:1196] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/3ee99a13e414454492151ac4053b12f2
I20260810 22:28:44.408488 4487 ts_tablet_manager.cc:1939] T 3ee99a13e414454492151ac4053b12f2 P ca5dce231a2c4852b2edad1aded895ad: Deleting consensus metadata
I20260810 22:28:44.409890 3838 catalog_manager.cc:4841] TS ca5dce231a2c4852b2edad1aded895ad (127.1.221.131:32827): tablet 3ee99a13e414454492151ac4053b12f2 (table test-table [id=cf0933565ab340a4b344f1bc70130bae]) successfully deleted
I20260810 22:28:44.421242 4438 raft_consensus.cc:482] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:28:44.421418 4438 raft_consensus.cc:504] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:44.422549 4438 leader_election.cc:288] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 880c64863b3a4e5d850578fb56f35bd8 (127.1.221.129:36241), 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130:41699)
I20260810 22:28:44.428206 4491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.422885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:41699 (local address 127.1.221.131:51305)
0810 22:28:44.423273 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:44.423287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:44.423407 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:44.423751 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:44.423754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:44.423786 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:44.424161 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.424169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.425056 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.425060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.426846 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.426854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.427743 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.427745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.427828 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.427849 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 22:28:44.427942 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 22:28:44.428022 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":498,"thread_start_us":105,"threads_started":1}
I20260810 22:28:44.428617 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.423033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:51305 (local address 127.1.221.130:41699)
0810 22:28:44.423198 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:44.423204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:44.423354 (+ 150us) server_negotiation.cc:407] Connection header received
0810 22:28:44.423558 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:44.423561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:44.423626 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:44.423699 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:44.424297 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.424899 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.426982 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.427618 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.427748 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.427806 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:44.427966 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 22:28:44.428049 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 22:28:44.429143 4196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3860edc1a1984509993646611e8cdef1" candidate_uuid: "ca5dce231a2c4852b2edad1aded895ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" is_pre_election: true
I20260810 22:28:44.429293 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.422803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36241 (local address 127.1.221.131:54325)
0810 22:28:44.423226 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:44.423243 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:44.423384 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:44.423820 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:44.423823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:44.423834 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:28:44.423993 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.424001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.425029 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.425035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.426842 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.426854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.428853 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.428857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.428967 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.428994 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:28:44.429069 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 22:28:44.429144 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":192,"thread_start_us":106,"threads_started":1}
I20260810 22:28:44.429315 4196 raft_consensus.cc:2454] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5dce231a2c4852b2edad1aded895ad in term 0.
I20260810 22:28:44.429351 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.423124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:54325 (local address 127.1.221.129:36241)
0810 22:28:44.423264 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:44.423268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:44.423345 (+ 77us) server_negotiation.cc:407] Connection header received
0810 22:28:44.423507 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:44.423510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:44.423562 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:44.423650 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:28:44.424097 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.424898 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.427867 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.428594 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.428726 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.428773 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:44.429126 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 22:28:44.429220 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 22:28:44.429850 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3860edc1a1984509993646611e8cdef1" candidate_uuid: "ca5dce231a2c4852b2edad1aded895ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "880c64863b3a4e5d850578fb56f35bd8" is_pre_election: true
I20260810 22:28:44.430007 4028 raft_consensus.cc:2454] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca5dce231a2c4852b2edad1aded895ad in term 0.
I20260810 22:28:44.430047 4277 leader_election.cc:302] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b222cb3f76548e1b4d5d37a60b8e002, ca5dce231a2c4852b2edad1aded895ad; no voters:
I20260810 22:28:44.430339 4438 raft_consensus.cc:2781] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:28:44.430411 4438 raft_consensus.cc:482] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:28:44.430455 4438 raft_consensus.cc:3037] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:44.431542 4438 raft_consensus.cc:504] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:44.431928 4438 leader_election.cc:288] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [CANDIDATE]: Term 1 election: Requested vote from peers 880c64863b3a4e5d850578fb56f35bd8 (127.1.221.129:36241), 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130:41699)
I20260810 22:28:44.432235 4028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3860edc1a1984509993646611e8cdef1" candidate_uuid: "ca5dce231a2c4852b2edad1aded895ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "880c64863b3a4e5d850578fb56f35bd8"
I20260810 22:28:44.432343 4028 raft_consensus.cc:3037] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:44.432299 4196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3860edc1a1984509993646611e8cdef1" candidate_uuid: "ca5dce231a2c4852b2edad1aded895ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b222cb3f76548e1b4d5d37a60b8e002"
I20260810 22:28:44.432412 4196 raft_consensus.cc:3037] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:44.433290 4196 raft_consensus.cc:2454] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5dce231a2c4852b2edad1aded895ad in term 1.
I20260810 22:28:44.433509 4028 raft_consensus.cc:2454] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca5dce231a2c4852b2edad1aded895ad in term 1.
I20260810 22:28:44.433619 4277 leader_election.cc:302] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b222cb3f76548e1b4d5d37a60b8e002, ca5dce231a2c4852b2edad1aded895ad; no voters:
I20260810 22:28:44.433806 4438 raft_consensus.cc:2781] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:44.434041 4438 raft_consensus.cc:686] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [term 1 LEADER]: Becoming Leader. State: Replica: ca5dce231a2c4852b2edad1aded895ad, State: Running, Role: LEADER
I20260810 22:28:44.434319 4438 consensus_queue.cc:237] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:44.436759 3855 catalog_manager.cc:5495] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ca5dce231a2c4852b2edad1aded895ad (127.1.221.131). New cstate: current_term: 1 leader_uuid: "ca5dce231a2c4852b2edad1aded895ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } health_report { overall_health: HEALTHY } } }
I20260810 22:28:44.459991 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.456406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.1.221.131:32827)
0810 22:28:44.456521 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:44.456524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:44.456537 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:28:44.456645 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:44.456647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:44.456702 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:44.456835 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:28:44.457082 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.457648 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.458486 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.458707 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.458820 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.459101 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:28:44.459566 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:28:44.459701 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:44.459782 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 22:28:44.459843 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 22:28:44.471285 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.467708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:53013 (local address 127.1.221.129:36241)
0810 22:28:44.467878 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:44.467883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:44.467899 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:28:44.467955 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:44.467957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:44.467998 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:44.468081 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:28:44.468722 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.469222 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.470165 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.470393 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.470501 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.470798 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:28:44.470870 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:28:44.470948 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:44.471075 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 22:28:44.471126 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 22:28:44.471305 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.467383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36241 (local address 127.1.221.131:53013)
0810 22:28:44.467689 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:44.467702 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:44.467825 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:44.468113 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:44.468116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:44.468123 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:28:44.468580 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.468587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.469339 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.469345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.470034 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.470043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.470515 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.470518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.470606 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.470649 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:28:44.470968 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:28:44.470976 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:28:44.471058 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 22:28:44.471136 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":109,"threads_started":1}
I20260810 22:28:44.478267 4194 raft_consensus.cc:1261] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Refusing update from remote peer ca5dce231a2c4852b2edad1aded895ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:44.478349 4028 raft_consensus.cc:1261] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Refusing update from remote peer ca5dce231a2c4852b2edad1aded895ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:44.478947 4438 consensus_queue.cc:1035] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:44.480927 4493 consensus_queue.cc:1035] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:44.504747 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.500704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:32827 (local address 127.1.221.129:45097)
0810 22:28:44.501007 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:44.501023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:44.501110 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:44.501616 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:44.501620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:44.501628 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:28:44.501739 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.501749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.502729 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.502733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.503328 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:44.503336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.504010 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.504014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:44.504099 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.504125 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:28:44.504447 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:28:44.504455 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:28:44.504513 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 22:28:44.504607 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260810 22:28:44.505095 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:44.501148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:45097 (local address 127.1.221.131:32827)
0810 22:28:44.501261 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:44.501266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:44.501279 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:28:44.501320 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:44.501323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:44.501376 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:44.501446 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:28:44.501870 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.502598 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.503455 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:44.503820 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:44.503928 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:44.504253 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:28:44.504338 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:28:44.504405 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:44.504916 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 22:28:44.504969 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 22:28:44.505738 4463 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 22:28:46.069402 3855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:28:46.069561 3855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:28:46.072127 3855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d: Sending DeleteTablet for 3 replicas of tablet 3860edc1a1984509993646611e8cdef1
I20260810 22:28:46.072765 4006 tablet_service.cc:1514] Processing DeleteTablet for tablet 3860edc1a1984509993646611e8cdef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:46 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:55990
I20260810 22:28:46.072928 4331 tablet_service.cc:1514] Processing DeleteTablet for tablet 3860edc1a1984509993646611e8cdef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:46 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:38368
I20260810 22:28:46.072950 4173 tablet_service.cc:1514] Processing DeleteTablet for tablet 3860edc1a1984509993646611e8cdef1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:46 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:60510
I20260810 22:28:46.073206 4526 tablet_replica.cc:330] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8: stopping tablet replica
I20260810 22:28:46.073372 4528 tablet_replica.cc:330] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002: stopping tablet replica
I20260810 22:28:46.073376 4526 raft_consensus.cc:2229] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:28:46.073486 4528 raft_consensus.cc:2229] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:28:46.073511 4526 raft_consensus.cc:2258] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:46.073632 4528 raft_consensus.cc:2258] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:46.074386 4526 ts_tablet_manager.cc:1905] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:46.074386 4528 ts_tablet_manager.cc:1905] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:46.077270 4528 ts_tablet_manager.cc:1918] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:28:46.077279 4526 ts_tablet_manager.cc:1918] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:28:46.077365 4526 log.cc:1196] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/wal/wals/3860edc1a1984509993646611e8cdef1
I20260810 22:28:46.077365 4528 log.cc:1196] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/3860edc1a1984509993646611e8cdef1
I20260810 22:28:46.077483 4527 tablet_replica.cc:330] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad: stopping tablet replica
I20260810 22:28:46.077688 4526 ts_tablet_manager.cc:1939] T 3860edc1a1984509993646611e8cdef1 P 880c64863b3a4e5d850578fb56f35bd8: Deleting consensus metadata
I20260810 22:28:46.077688 4528 ts_tablet_manager.cc:1939] T 3860edc1a1984509993646611e8cdef1 P 0b222cb3f76548e1b4d5d37a60b8e002: Deleting consensus metadata
I20260810 22:28:46.077714 4527 raft_consensus.cc:2229] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:28:46.078228 4527 raft_consensus.cc:2258] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:46.078550 3840 catalog_manager.cc:4841] TS 880c64863b3a4e5d850578fb56f35bd8 (127.1.221.129:36241): tablet 3860edc1a1984509993646611e8cdef1 (table test-table [id=7338312f9d204019abd3257356dc8d92]) successfully deleted
I20260810 22:28:46.078552 3838 catalog_manager.cc:4841] TS 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130:41699): tablet 3860edc1a1984509993646611e8cdef1 (table test-table [id=7338312f9d204019abd3257356dc8d92]) successfully deleted
I20260810 22:28:46.079185 4527 ts_tablet_manager.cc:1905] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:46.082069 4527 ts_tablet_manager.cc:1918] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:28:46.082139 4527 log.cc:1196] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/3860edc1a1984509993646611e8cdef1
I20260810 22:28:46.082430 4527 ts_tablet_manager.cc:1939] T 3860edc1a1984509993646611e8cdef1 P ca5dce231a2c4852b2edad1aded895ad: Deleting consensus metadata
I20260810 22:28:46.083218 3838 catalog_manager.cc:4841] TS ca5dce231a2c4852b2edad1aded895ad (127.1.221.131:32827): tablet 3860edc1a1984509993646611e8cdef1 (table test-table [id=7338312f9d204019abd3257356dc8d92]) successfully deleted
Aug 10 22:28:46 dist-test-slave-njxd krb5kdc[3806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:28:46 dist-test-slave-njxd krb5kdc[3806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400926, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:46.107435 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:46.094666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.1.221.190:37029)
0810 22:28:46.094961 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:46.094972 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:46.095049 (+ 77us) server_negotiation.cc:407] Connection header received
0810 22:28:46.095253 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:46.095256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:46.095305 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:46.095401 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:46.096134 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:46.096797 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:46.097722 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:46.097961 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:46.098074 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:46.104386 (+ 6312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:46.104389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:46.104391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:46.104423 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:46.105842 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:46.106317 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:46.106321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:46.106322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:46.106415 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:46.106667 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:46.106669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:46.106670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:46.106864 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:46.106972 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:46.107179 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 22:28:46.107295 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
I20260810 22:28:46.110196 3855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:28:46.110698 3855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:28:46.115291 4006 tablet_service.cc:1467] Processing CreateTablet for tablet 34b7c1f6477744778dab3732689e82a0 (DEFAULT_TABLE table=test-table [id=0d42d7871da242119e7b46265c88ea0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:46.115365 4173 tablet_service.cc:1467] Processing CreateTablet for tablet 34b7c1f6477744778dab3732689e82a0 (DEFAULT_TABLE table=test-table [id=0d42d7871da242119e7b46265c88ea0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:46.115446 4331 tablet_service.cc:1467] Processing CreateTablet for tablet 34b7c1f6477744778dab3732689e82a0 (DEFAULT_TABLE table=test-table [id=0d42d7871da242119e7b46265c88ea0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:46.115641 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b7c1f6477744778dab3732689e82a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:46.115680 4173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b7c1f6477744778dab3732689e82a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:46.115732 4331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34b7c1f6477744778dab3732689e82a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:46.118057 4537 tablet_bootstrap.cc:492] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002: Bootstrap starting.
I20260810 22:28:46.118553 4538 tablet_bootstrap.cc:492] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad: Bootstrap starting.
I20260810 22:28:46.119306 4537 tablet_bootstrap.cc:654] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:46.119464 4538 tablet_bootstrap.cc:654] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:46.120049 4539 tablet_bootstrap.cc:492] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8: Bootstrap starting.
I20260810 22:28:46.120498 4537 tablet_bootstrap.cc:492] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002: No bootstrap required, opened a new log
I20260810 22:28:46.120569 4537 ts_tablet_manager.cc:1397] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:28:46.120908 4539 tablet_bootstrap.cc:654] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:46.121057 4537 raft_consensus.cc:348] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.121176 4537 raft_consensus.cc:374] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:46.121212 4537 raft_consensus.cc:729] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b222cb3f76548e1b4d5d37a60b8e002, State: Initialized, Role: FOLLOWER
I20260810 22:28:46.121351 4537 consensus_queue.cc:260] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.121821 4537 ts_tablet_manager.cc:1428] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:28:46.122007 4538 tablet_bootstrap.cc:492] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad: No bootstrap required, opened a new log
I20260810 22:28:46.122104 4538 ts_tablet_manager.cc:1397] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 22:28:46.122525 4538 raft_consensus.cc:348] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.122655 4538 raft_consensus.cc:374] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:46.122706 4538 raft_consensus.cc:729] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5dce231a2c4852b2edad1aded895ad, State: Initialized, Role: FOLLOWER
I20260810 22:28:46.122836 4538 consensus_queue.cc:260] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.123100 4538 ts_tablet_manager.cc:1428] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:28:46.123584 4539 tablet_bootstrap.cc:492] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8: No bootstrap required, opened a new log
I20260810 22:28:46.123716 4539 ts_tablet_manager.cc:1397] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 22:28:46.124215 4539 raft_consensus.cc:348] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.124354 4539 raft_consensus.cc:374] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:46.124393 4539 raft_consensus.cc:729] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880c64863b3a4e5d850578fb56f35bd8, State: Initialized, Role: FOLLOWER
I20260810 22:28:46.124531 4539 consensus_queue.cc:260] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.124809 4539 ts_tablet_manager.cc:1428] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:28:46.127825 4510 raft_consensus.cc:482] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:28:46.127907 4510 raft_consensus.cc:504] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.128229 4510 leader_election.cc:288] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130:41699), ca5dce231a2c4852b2edad1aded895ad (127.1.221.131:32827)
I20260810 22:28:46.128553 4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7c1f6477744778dab3732689e82a0" candidate_uuid: "880c64863b3a4e5d850578fb56f35bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" is_pre_election: true
I20260810 22:28:46.128590 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7c1f6477744778dab3732689e82a0" candidate_uuid: "880c64863b3a4e5d850578fb56f35bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5dce231a2c4852b2edad1aded895ad" is_pre_election: true
I20260810 22:28:46.128697 4194 raft_consensus.cc:2454] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 880c64863b3a4e5d850578fb56f35bd8 in term 0.
I20260810 22:28:46.128716 4352 raft_consensus.cc:2454] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 880c64863b3a4e5d850578fb56f35bd8 in term 0.
I20260810 22:28:46.129007 3954 leader_election.cc:302] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b222cb3f76548e1b4d5d37a60b8e002, 880c64863b3a4e5d850578fb56f35bd8; no voters:
I20260810 22:28:46.129184 4510 raft_consensus.cc:2781] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:28:46.129241 4510 raft_consensus.cc:482] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:28:46.129292 4510 raft_consensus.cc:3037] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:46.130306 4510 raft_consensus.cc:504] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.130622 4510 leader_election.cc:288] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130:41699), ca5dce231a2c4852b2edad1aded895ad (127.1.221.131:32827)
I20260810 22:28:46.131001 4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7c1f6477744778dab3732689e82a0" candidate_uuid: "880c64863b3a4e5d850578fb56f35bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b222cb3f76548e1b4d5d37a60b8e002"
I20260810 22:28:46.131047 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34b7c1f6477744778dab3732689e82a0" candidate_uuid: "880c64863b3a4e5d850578fb56f35bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5dce231a2c4852b2edad1aded895ad"
I20260810 22:28:46.131150 4194 raft_consensus.cc:3037] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:46.131196 4352 raft_consensus.cc:3037] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:46.132252 4194 raft_consensus.cc:2454] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 880c64863b3a4e5d850578fb56f35bd8 in term 1.
I20260810 22:28:46.132321 4352 raft_consensus.cc:2454] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 880c64863b3a4e5d850578fb56f35bd8 in term 1.
I20260810 22:28:46.132561 3954 leader_election.cc:302] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b222cb3f76548e1b4d5d37a60b8e002, 880c64863b3a4e5d850578fb56f35bd8; no voters:
I20260810 22:28:46.132736 4510 raft_consensus.cc:2781] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:46.132828 4510 raft_consensus.cc:686] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 LEADER]: Becoming Leader. State: Replica: 880c64863b3a4e5d850578fb56f35bd8, State: Running, Role: LEADER
I20260810 22:28:46.132968 4510 consensus_queue.cc:237] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.134464 3855 catalog_manager.cc:5495] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 880c64863b3a4e5d850578fb56f35bd8 (127.1.221.129). New cstate: current_term: 1 leader_uuid: "880c64863b3a4e5d850578fb56f35bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } health_report { overall_health: UNKNOWN } } }
I20260810 22:28:46.154721 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:46.151302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.1.221.129:36241)
0810 22:28:46.151601 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:46.151609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:46.151622 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:28:46.151661 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:46.151663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:46.151705 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:46.151776 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:28:46.151993 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:46.152527 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:46.153495 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:46.153841 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:46.153927 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:46.154168 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:28:46.154247 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:28:46.154305 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:46.154447 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 22:28:46.154498 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":156,"threads_started":1}
I20260810 22:28:46.163746 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:46.160852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:60151 (local address 127.1.221.129:36241)
0810 22:28:46.160983 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:46.160987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:46.161106 (+ 119us) server_negotiation.cc:407] Connection header received
0810 22:28:46.161208 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:46.161210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:46.161253 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:46.161315 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:28:46.161523 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:46.162023 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:46.162839 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:46.163045 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:46.163128 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:46.163370 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:28:46.163441 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:28:46.163497 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:46.163573 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 22:28:46.163623 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:28:46.163761 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:46.160702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36241 (local address 127.1.221.129:60151)
0810 22:28:46.161022 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:46.161036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:46.161126 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:46.161337 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:46.161339 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:46.161345 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:28:46.161432 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:46.161436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:46.162121 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:46.162123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:46.162737 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:46.162745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:46.163139 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:46.163142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:46.163203 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:46.163229 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:28:46.163512 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:28:46.163517 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 22:28:46.163565 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 22:28:46.163635 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":139,"threads_started":1}
I20260810 22:28:46.169881 4352 raft_consensus.cc:1261] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Refusing update from remote peer 880c64863b3a4e5d850578fb56f35bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:46.169924 4194 raft_consensus.cc:1261] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Refusing update from remote peer 880c64863b3a4e5d850578fb56f35bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:46.170288 4504 consensus_queue.cc:1035] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:46.170452 4510 consensus_queue.cc:1035] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:46.205314 3855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:28:46.205503 3855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:28:46.207609 3855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d: Sending DeleteTablet for 3 replicas of tablet 34b7c1f6477744778dab3732689e82a0
I20260810 22:28:46.208165 4006 tablet_service.cc:1514] Processing DeleteTablet for tablet 34b7c1f6477744778dab3732689e82a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:46 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:55990
I20260810 22:28:46.208365 4526 tablet_replica.cc:330] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8: stopping tablet replica
I20260810 22:28:46.208496 4526 raft_consensus.cc:2229] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:28:46.208556 4331 tablet_service.cc:1514] Processing DeleteTablet for tablet 34b7c1f6477744778dab3732689e82a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:46 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:38368
I20260810 22:28:46.208556 4173 tablet_service.cc:1514] Processing DeleteTablet for tablet 34b7c1f6477744778dab3732689e82a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:46 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:60510
I20260810 22:28:46.208789 4527 tablet_replica.cc:330] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad: stopping tablet replica
I20260810 22:28:46.208818 4526 raft_consensus.cc:2258] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:46.208911 4527 raft_consensus.cc:2229] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:28:46.209031 4527 raft_consensus.cc:2258] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:46.209221 4528 tablet_replica.cc:330] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002: stopping tablet replica
I20260810 22:28:46.209362 4528 raft_consensus.cc:2229] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:28:46.209487 4528 raft_consensus.cc:2258] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:46.209781 4526 ts_tablet_manager.cc:1905] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:46.210348 4527 ts_tablet_manager.cc:1905] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:46.211084 4528 ts_tablet_manager.cc:1905] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:46.212646 4526 ts_tablet_manager.cc:1918] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:28:46.212721 4526 log.cc:1196] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-0/wal/wals/34b7c1f6477744778dab3732689e82a0
I20260810 22:28:46.213020 4526 ts_tablet_manager.cc:1939] T 34b7c1f6477744778dab3732689e82a0 P 880c64863b3a4e5d850578fb56f35bd8: Deleting consensus metadata
I20260810 22:28:46.213081 4527 ts_tablet_manager.cc:1918] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:28:46.213152 4527 log.cc:1196] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/34b7c1f6477744778dab3732689e82a0
I20260810 22:28:46.213433 4527 ts_tablet_manager.cc:1939] T 34b7c1f6477744778dab3732689e82a0 P ca5dce231a2c4852b2edad1aded895ad: Deleting consensus metadata
I20260810 22:28:46.213527 4528 ts_tablet_manager.cc:1918] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:28:46.213629 4528 log.cc:1196] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/34b7c1f6477744778dab3732689e82a0
I20260810 22:28:46.213786 3840 catalog_manager.cc:4841] TS 880c64863b3a4e5d850578fb56f35bd8 (127.1.221.129:36241): tablet 34b7c1f6477744778dab3732689e82a0 (table test-table [id=0d42d7871da242119e7b46265c88ea0c]) successfully deleted
I20260810 22:28:46.213891 4528 ts_tablet_manager.cc:1939] T 34b7c1f6477744778dab3732689e82a0 P 0b222cb3f76548e1b4d5d37a60b8e002: Deleting consensus metadata
I20260810 22:28:46.214001 3855 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:28:46.214409 3855 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:28:46.214596 3838 catalog_manager.cc:4841] TS 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130:41699): tablet 34b7c1f6477744778dab3732689e82a0 (table test-table [id=0d42d7871da242119e7b46265c88ea0c]) successfully deleted
I20260810 22:28:46.215986 3838 catalog_manager.cc:4841] TS ca5dce231a2c4852b2edad1aded895ad (127.1.221.131:32827): tablet 34b7c1f6477744778dab3732689e82a0 (table test-table [id=0d42d7871da242119e7b46265c88ea0c]) successfully deleted
I20260810 22:28:46.219141 4006 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3963bfb6254b6c86c689c1a33fa972 (DEFAULT_TABLE table=test-table [id=455927286453449baf0fff2edf6e57b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:46.219309 4331 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3963bfb6254b6c86c689c1a33fa972 (DEFAULT_TABLE table=test-table [id=455927286453449baf0fff2edf6e57b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:46.219336 4173 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3963bfb6254b6c86c689c1a33fa972 (DEFAULT_TABLE table=test-table [id=455927286453449baf0fff2edf6e57b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:28:46.219419 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3963bfb6254b6c86c689c1a33fa972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:46.219538 4331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3963bfb6254b6c86c689c1a33fa972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:46.219594 4173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3963bfb6254b6c86c689c1a33fa972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:46.221731 4538 tablet_bootstrap.cc:492] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad: Bootstrap starting.
I20260810 22:28:46.222031 4539 tablet_bootstrap.cc:492] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8: Bootstrap starting.
I20260810 22:28:46.222600 4538 tablet_bootstrap.cc:654] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:46.222898 4539 tablet_bootstrap.cc:654] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:46.223157 4537 tablet_bootstrap.cc:492] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002: Bootstrap starting.
I20260810 22:28:46.223999 4537 tablet_bootstrap.cc:654] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:46.224931 4539 tablet_bootstrap.cc:492] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8: No bootstrap required, opened a new log
I20260810 22:28:46.225020 4539 ts_tablet_manager.cc:1397] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:28:46.225059 4537 tablet_bootstrap.cc:492] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002: No bootstrap required, opened a new log
I20260810 22:28:46.225135 4537 ts_tablet_manager.cc:1397] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 22:28:46.225296 4538 tablet_bootstrap.cc:492] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad: No bootstrap required, opened a new log
I20260810 22:28:46.225375 4538 ts_tablet_manager.cc:1397] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 22:28:46.225448 4539 raft_consensus.cc:348] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.225517 4537 raft_consensus.cc:348] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.225656 4537 raft_consensus.cc:374] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:46.225692 4537 raft_consensus.cc:729] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b222cb3f76548e1b4d5d37a60b8e002, State: Initialized, Role: FOLLOWER
I20260810 22:28:46.225553 4539 raft_consensus.cc:374] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:46.225831 4539 raft_consensus.cc:729] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880c64863b3a4e5d850578fb56f35bd8, State: Initialized, Role: FOLLOWER
I20260810 22:28:46.225814 4537 consensus_queue.cc:260] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.225807 4538 raft_consensus.cc:348] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.225919 4538 raft_consensus.cc:374] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:46.225950 4538 raft_consensus.cc:729] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca5dce231a2c4852b2edad1aded895ad, State: Initialized, Role: FOLLOWER
I20260810 22:28:46.225965 4539 consensus_queue.cc:260] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.226060 4537 ts_tablet_manager.cc:1428] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:28:46.226060 4538 consensus_queue.cc:260] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.226234 4539 ts_tablet_manager.cc:1428] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:28:46.226305 4538 ts_tablet_manager.cc:1428] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:28:46.586596 4469 raft_consensus.cc:482] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:28:46.586797 4469 raft_consensus.cc:504] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.587286 4469 leader_election.cc:288] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130:41699), ca5dce231a2c4852b2edad1aded895ad (127.1.221.131:32827)
I20260810 22:28:46.587796 4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3963bfb6254b6c86c689c1a33fa972" candidate_uuid: "880c64863b3a4e5d850578fb56f35bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" is_pre_election: true
I20260810 22:28:46.588095 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3963bfb6254b6c86c689c1a33fa972" candidate_uuid: "880c64863b3a4e5d850578fb56f35bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5dce231a2c4852b2edad1aded895ad" is_pre_election: true
I20260810 22:28:46.588271 4351 raft_consensus.cc:2454] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 880c64863b3a4e5d850578fb56f35bd8 in term 0.
I20260810 22:28:46.587973 4194 raft_consensus.cc:2454] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 880c64863b3a4e5d850578fb56f35bd8 in term 0.
I20260810 22:28:46.588653 3954 leader_election.cc:302] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 880c64863b3a4e5d850578fb56f35bd8, ca5dce231a2c4852b2edad1aded895ad; no voters:
I20260810 22:28:46.588858 4469 raft_consensus.cc:2781] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:28:46.588940 4469 raft_consensus.cc:482] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:28:46.588997 4469 raft_consensus.cc:3037] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:46.590235 4469 raft_consensus.cc:504] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.590574 4469 leader_election.cc:288] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130:41699), ca5dce231a2c4852b2edad1aded895ad (127.1.221.131:32827)
I20260810 22:28:46.590930 4194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3963bfb6254b6c86c689c1a33fa972" candidate_uuid: "880c64863b3a4e5d850578fb56f35bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b222cb3f76548e1b4d5d37a60b8e002"
I20260810 22:28:46.591085 4194 raft_consensus.cc:3037] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:46.591027 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3963bfb6254b6c86c689c1a33fa972" candidate_uuid: "880c64863b3a4e5d850578fb56f35bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca5dce231a2c4852b2edad1aded895ad"
I20260810 22:28:46.591178 4351 raft_consensus.cc:3037] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:46.592123 4194 raft_consensus.cc:2454] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 880c64863b3a4e5d850578fb56f35bd8 in term 1.
I20260810 22:28:46.592193 4351 raft_consensus.cc:2454] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 880c64863b3a4e5d850578fb56f35bd8 in term 1.
I20260810 22:28:46.592404 3954 leader_election.cc:302] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b222cb3f76548e1b4d5d37a60b8e002, 880c64863b3a4e5d850578fb56f35bd8; no voters:
I20260810 22:28:46.592581 4469 raft_consensus.cc:2781] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:46.592695 4469 raft_consensus.cc:686] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [term 1 LEADER]: Becoming Leader. State: Replica: 880c64863b3a4e5d850578fb56f35bd8, State: Running, Role: LEADER
I20260810 22:28:46.592859 4469 consensus_queue.cc:237] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } }
I20260810 22:28:46.594596 3855 catalog_manager.cc:5495] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 880c64863b3a4e5d850578fb56f35bd8 (127.1.221.129). New cstate: current_term: 1 leader_uuid: "880c64863b3a4e5d850578fb56f35bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "880c64863b3a4e5d850578fb56f35bd8" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 } health_report { overall_health: UNKNOWN } } }
I20260810 22:28:46.691174 4351 raft_consensus.cc:1261] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Refusing update from remote peer 880c64863b3a4e5d850578fb56f35bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:46.691397 4194 raft_consensus.cc:1261] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Refusing update from remote peer 880c64863b3a4e5d850578fb56f35bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:28:46.691587 4504 consensus_queue.cc:1035] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca5dce231a2c4852b2edad1aded895ad" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:28:46.691802 4510 consensus_queue.cc:1035] T 8d3963bfb6254b6c86c689c1a33fa972 P 880c64863b3a4e5d850578fb56f35bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b222cb3f76548e1b4d5d37a60b8e002" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:28:46.705690 4502 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 22:28:48.262826 3852 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:28:48.262981 3852 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:28:48.265471 3852 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5290a0e324214d519b2655124f48d94d: Sending DeleteTablet for 3 replicas of tablet 8d3963bfb6254b6c86c689c1a33fa972
I20260810 22:28:48.266184 4173 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3963bfb6254b6c86c689c1a33fa972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:48 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:60510
I20260810 22:28:48.266256 4331 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3963bfb6254b6c86c689c1a33fa972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:48 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:38368
I20260810 22:28:48.266554 4586 tablet_replica.cc:330] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002: stopping tablet replica
I20260810 22:28:48.266568 4005 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d3963bfb6254b6c86c689c1a33fa972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:28:48 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:55990
I20260810 22:28:48.266628 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 3935
I20260810 22:28:48.269060 4586 raft_consensus.cc:2229] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:28:48.269254 4586 raft_consensus.cc:2258] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:48.269946 4586 ts_tablet_manager.cc:1905] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:28:48.272379 4586 ts_tablet_manager.cc:1918] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:28:48.272485 4586 log.cc:1196] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/8d3963bfb6254b6c86c689c1a33fa972
I20260810 22:28:48.272744 4586 ts_tablet_manager.cc:1939] T 8d3963bfb6254b6c86c689c1a33fa972 P 0b222cb3f76548e1b4d5d37a60b8e002: Deleting consensus metadata
I20260810 22:28:48.273499 3838 catalog_manager.cc:4841] TS 0b222cb3f76548e1b4d5d37a60b8e002 (127.1.221.130:41699): tablet 8d3963bfb6254b6c86c689c1a33fa972 (table test-table [id=455927286453449baf0fff2edf6e57b4]) successfully deleted
I20260810 22:28:48.277812 4587 tablet_replica.cc:330] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad: stopping tablet replica
I20260810 22:28:48.278029 4587 raft_consensus.cc:2229] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:28:48.278205 4587 raft_consensus.cc:2258] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:28:48.279043 4587 ts_tablet_manager.cc:1905] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 22:28:48.279145 3840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:36241 (error 108)
I20260810 22:28:48.279338 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 4091
I20260810 22:28:48.283113 4589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:48.282369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36241 (local address 127.0.0.1:56004)
0810 22:28:48.282945 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:48.283021 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:36241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":97,"threads_started":1}
W20260810 22:28:48.283392 3840 catalog_manager.cc:4584] TS 880c64863b3a4e5d850578fb56f35bd8 (127.1.221.129:36241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d3963bfb6254b6c86c689c1a33fa972: Network error: Client connection negotiation failed: client connection to 127.1.221.129:36241: connect: Connection refused (error 111)
I20260810 22:28:48.285244 4587 ts_tablet_manager.cc:1918] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:28:48.285342 4587 log.cc:1196] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/8d3963bfb6254b6c86c689c1a33fa972
I20260810 22:28:48.285717 4587 ts_tablet_manager.cc:1939] T 8d3963bfb6254b6c86c689c1a33fa972 P ca5dce231a2c4852b2edad1aded895ad: Deleting consensus metadata
I20260810 22:28:48.286587 3838 catalog_manager.cc:4841] TS ca5dce231a2c4852b2edad1aded895ad (127.1.221.131:32827): tablet 8d3963bfb6254b6c86c689c1a33fa972 (table test-table [id=455927286453449baf0fff2edf6e57b4]) successfully deleted
I20260810 22:28:48.290845 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 4258
I20260810 22:28:48.299026 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 3819
2026-08-10T22:28:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6769 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:28:48 dist-test-slave-njxd krb5kdc[4592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:28:48 dist-test-slave-njxd krb5kdc[4592](info): set up 2 sockets
Aug 10 22:28:48 dist-test-slave-njxd krb5kdc[4592](info): commencing operation
krb5kdc: starting...
W20260810 22:28:49.404419 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260810 22:28:49.673583 1910 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.179s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/test-user.keytab.
W20260810 22:28:50.295104 1910 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.621s user 0.002s sys 0.000s
Aug 10 22:28:50 dist-test-slave-njxd krb5kdc[4592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:28:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:28:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
W20260810 22:28:51.140161 1910 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.221.190: real 0.818s user 0.000s sys 0.004s
I20260810 22:28:51.140872 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:37457
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:37457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:51.250873 4607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:51.251163 4607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:51.251238 4607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:51.255373 4607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:28:51.255470 4607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:51.255507 4607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:51.255568 4607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:28:51.255604 4607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:28:51.260962 4607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:37457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:37457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:51.262167 4607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:51.263243 4607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:28:51.272030 4607 server_base.cc:1034] running on GCE node
W20260810 22:28:51.271973 4614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:51.271979 4613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:51.271955 4616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:51.272746 4607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:51.273881 4607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:51.275069 4607 hybrid_clock.cc:648] HybridClock initialized: now 1786400931275055 us; error 25 us; skew 500 ppm
Aug 10 22:28:51 dist-test-slave-njxd krb5kdc[4592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400931, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:51.280220 4607 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:28:51.281068 4607 webserver.cc:460] Webserver started at http://127.1.221.190:41745/ using document root <none> and password file <none>
I20260810 22:28:51.281659 4607 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:51.281746 4607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:51.281975 4607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:51.283981 4607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "626933a63322467e918c5ae6e70d0cd6"
format_stamp: "Formatted at 2026-08-10 22:28:51 on dist-test-slave-njxd"
server_key: "b8df2a4609866c48127f941c3ac8b743"
server_key_iv: "ac13b7670a32f292c768b055c16d4853"
server_key_version: "encryptionkey@0"
I20260810 22:28:51.284545 4607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "626933a63322467e918c5ae6e70d0cd6"
format_stamp: "Formatted at 2026-08-10 22:28:51 on dist-test-slave-njxd"
server_key: "b8df2a4609866c48127f941c3ac8b743"
server_key_iv: "ac13b7670a32f292c768b055c16d4853"
server_key_version: "encryptionkey@0"
I20260810 22:28:51.288141 4607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 22:28:51.290535 4624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:51.291783 4607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:28:51.291927 4607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "626933a63322467e918c5ae6e70d0cd6"
format_stamp: "Formatted at 2026-08-10 22:28:51 on dist-test-slave-njxd"
server_key: "b8df2a4609866c48127f941c3ac8b743"
server_key_iv: "ac13b7670a32f292c768b055c16d4853"
server_key_version: "encryptionkey@0"
I20260810 22:28:51.292037 4607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:51.307587 4607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:51.311021 4607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:51.311226 4607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:51.318252 4607 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:37457
I20260810 22:28:51.318295 4682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:37457 every 8 connection(s)
I20260810 22:28:51.319186 4607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:28:51.321969 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:51.327307 4683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6: Bootstrap starting.
I20260810 22:28:51.327862 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 4607
I20260810 22:28:51.327975 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:28:51.328256 1910 external_mini_cluster.cc:1428] Setting key 92f5006c23ac46623855be3610e29d69
I20260810 22:28:51.330163 4683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:51.330894 4683 log.cc:824] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:51.332650 4683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6: No bootstrap required, opened a new log
I20260810 22:28:51.335944 4683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626933a63322467e918c5ae6e70d0cd6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37457 } }
I20260810 22:28:51.336186 4683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:51.336269 4683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626933a63322467e918c5ae6e70d0cd6, State: Initialized, Role: FOLLOWER
I20260810 22:28:51.336767 4683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626933a63322467e918c5ae6e70d0cd6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37457 } }
Aug 10 22:28:51 dist-test-slave-njxd krb5kdc[4592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:51.336931 4683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:28:51.337015 4683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:28:51.337141 4683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:51.338128 4683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626933a63322467e918c5ae6e70d0cd6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37457 } }
I20260810 22:28:51.338500 4683 leader_election.cc:302] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 626933a63322467e918c5ae6e70d0cd6; no voters:
I20260810 22:28:51.338872 4683 leader_election.cc:288] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:28:51.338989 4688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:51.339234 4688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [term 1 LEADER]: Becoming Leader. State: Replica: 626933a63322467e918c5ae6e70d0cd6, State: Running, Role: LEADER
I20260810 22:28:51.339591 4688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626933a63322467e918c5ae6e70d0cd6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37457 } }
I20260810 22:28:51.340185 4683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:28:51.340775 4690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 626933a63322467e918c5ae6e70d0cd6. Latest consensus state: current_term: 1 leader_uuid: "626933a63322467e918c5ae6e70d0cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626933a63322467e918c5ae6e70d0cd6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37457 } } }
I20260810 22:28:51.341555 4690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:51.341434 4689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "626933a63322467e918c5ae6e70d0cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626933a63322467e918c5ae6e70d0cd6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37457 } } }
I20260810 22:28:51.341909 4689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6 [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:51.342219 4697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:28:51.343459 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:51.329658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.1.221.190:37457)
0810 22:28:51.330076 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:51.330091 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:51.330127 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:28:51.330795 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:51.330813 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:51.331169 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:51.331479 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:51.332263 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:51.333246 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:51.334135 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:51.334618 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:51.334797 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:51.337562 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:51.337608 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:51.337627 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:51.337657 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:51.339831 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:51.340376 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:51.340382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:51.340388 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:51.340528 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:51.340864 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:51.340868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:51.340870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:51.341460 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:51.341652 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:51.342188 (+ 536us) server_negotiation.cc:299] Negotiation successful
0810 22:28:51.342500 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":119,"threads_started":1}
I20260810 22:28:51.346086 4697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:28:51.351728 4697 catalog_manager.cc:1347] Generated new cluster ID: cea0a8eb8f754638900d6210b86ecb80
I20260810 22:28:51.351805 4697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:28:51.373910 4697 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:28:51.375291 4697 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:28:51.382390 4697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 626933a63322467e918c5ae6e70d0cd6: Generated new TSK 0
I20260810 22:28:51.383198 4697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:28:51.418843 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37457
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:51.534858 4713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:51.535121 4713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:51.535167 4713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:51.539244 4713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:51.539304 4713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:51.539394 4713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:28:51.544982 4713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:51.546244 4713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:51.547312 4713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:28:51.557322 4713 server_base.cc:1034] running on GCE node
W20260810 22:28:51.557281 4720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:51.557271 4723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:51.557264 4721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:51.558084 4713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:51.558750 4713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:51.559924 4713 hybrid_clock.cc:648] HybridClock initialized: now 1786400931559891 us; error 45 us; skew 500 ppm
Aug 10 22:28:51 dist-test-slave-njxd krb5kdc[4592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400931, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:51.562918 4713 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:28:51.563828 4713 webserver.cc:460] Webserver started at http://127.1.221.129:42257/ using document root <none> and password file <none>
I20260810 22:28:51.564424 4713 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:51.564517 4713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:51.564760 4713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:51.566972 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "375d01fefa5b4977871859e9b8f06e8d"
format_stamp: "Formatted at 2026-08-10 22:28:51 on dist-test-slave-njxd"
server_key: "72a8972200dc5395f333d7948ffcbb31"
server_key_iv: "d11695b912fbfbbd3ee1dad720d08b7a"
server_key_version: "encryptionkey@0"
I20260810 22:28:51.567581 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "375d01fefa5b4977871859e9b8f06e8d"
format_stamp: "Formatted at 2026-08-10 22:28:51 on dist-test-slave-njxd"
server_key: "72a8972200dc5395f333d7948ffcbb31"
server_key_iv: "d11695b912fbfbbd3ee1dad720d08b7a"
server_key_version: "encryptionkey@0"
I20260810 22:28:51.571637 4713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:28:51.574296 4731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:51.575546 4713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:28:51.575690 4713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "375d01fefa5b4977871859e9b8f06e8d"
format_stamp: "Formatted at 2026-08-10 22:28:51 on dist-test-slave-njxd"
server_key: "72a8972200dc5395f333d7948ffcbb31"
server_key_iv: "d11695b912fbfbbd3ee1dad720d08b7a"
server_key_version: "encryptionkey@0"
I20260810 22:28:51.575812 4713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:51.610536 4713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:51.614087 4713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:51.614334 4713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:51.615041 4713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:51.615973 4713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:51.616050 4713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:51.616128 4713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:51.616171 4713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:51.625543 4713 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:36061
I20260810 22:28:51.625586 4862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:36061 every 8 connection(s)
I20260810 22:28:51.626574 4713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
Aug 10 22:28:51 dist-test-slave-njxd krb5kdc[4592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400931, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:51.636411 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 4713
I20260810 22:28:51.636539 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:28:51.636840 1910 external_mini_cluster.cc:1428] Setting key 5882bd082af679bfd919fdbea5d6911b
I20260810 22:28:51.640939 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:51.628484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:43307 (local address 127.1.221.190:37457)
0810 22:28:51.628686 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:51.628691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:51.629601 (+ 910us) server_negotiation.cc:407] Connection header received
0810 22:28:51.631009 (+ 1408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:51.631012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:51.631062 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:51.631151 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:51.632361 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:51.632901 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:51.634100 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:51.634336 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:51.634418 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:51.637072 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:51.637075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:51.637077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:51.637105 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:51.638662 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:51.639203 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:51.639208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:51.639210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:51.639304 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:51.639663 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:51.639668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:51.639670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:51.639948 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:51.640060 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:51.640583 (+ 523us) server_negotiation.cc:299] Negotiation successful
0810 22:28:51.640717 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 22:28:51.642004 4867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:51.628907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37457 (local address 127.1.221.129:43307)
0810 22:28:51.629408 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:51.629455 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:51.630796 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:51.631326 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:51.631333 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:51.631703 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:51.632220 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:51.632233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:51.633011 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:51.633014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:51.633989 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:51.633999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:51.634474 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:51.634476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:51.634627 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:51.635199 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:51.635217 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:51.636855 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:51.638796 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:51.638802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:51.638819 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:51.639092 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:51.639404 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:51.639407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:51.639409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:51.639564 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:51.640060 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:51.640069 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:51.640337 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 22:28:51.640538 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 22:28:51.640933 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":143,"threads_started":1}
I20260810 22:28:51.643333 4863 heartbeater.cc:347] Connected to a master server at 127.1.221.190:37457
I20260810 22:28:51.643602 4863 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:51.644132 4863 heartbeater.cc:511] Master 127.1.221.190:37457 requested a full tablet report, sending...
I20260810 22:28:51.645974 4643 ts_manager.cc:194] Registered new tserver with Master: 375d01fefa5b4977871859e9b8f06e8d (127.1.221.129:36061)
I20260810 22:28:51.647720 4643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:43307
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:28:51.668318 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37457
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:51.787408 4870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:51.787696 4870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:51.787743 4870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:51.792095 4870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:51.792165 4870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:51.792279 4870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:28:51.798383 4870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:51.799533 4870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:51.800642 4870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:51.811018 4875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:51.811018 4878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:51.811067 4876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:51.811322 4870 server_base.cc:1034] running on GCE node
I20260810 22:28:51.811746 4870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:51.812356 4870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:51.813524 4870 hybrid_clock.cc:648] HybridClock initialized: now 1786400931813509 us; error 25 us; skew 500 ppm
Aug 10 22:28:51 dist-test-slave-njxd krb5kdc[4592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400931, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:51.816344 4870 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:28:51.817209 4870 webserver.cc:460] Webserver started at http://127.1.221.130:46603/ using document root <none> and password file <none>
I20260810 22:28:51.817863 4870 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:51.817960 4870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:51.818200 4870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:51.820274 4870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "0a8ed05e9c6b44fca9ff4b70f5c44333"
format_stamp: "Formatted at 2026-08-10 22:28:51 on dist-test-slave-njxd"
server_key: "cfad59d5ea2db1adf70dabf7202860a0"
server_key_iv: "cfb7a58a8b7cb42f8b89754fb060ef83"
server_key_version: "encryptionkey@0"
I20260810 22:28:51.821091 4870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "0a8ed05e9c6b44fca9ff4b70f5c44333"
format_stamp: "Formatted at 2026-08-10 22:28:51 on dist-test-slave-njxd"
server_key: "cfad59d5ea2db1adf70dabf7202860a0"
server_key_iv: "cfb7a58a8b7cb42f8b89754fb060ef83"
server_key_version: "encryptionkey@0"
I20260810 22:28:51.825090 4870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:28:51.827692 4888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:51.828917 4870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:28:51.829056 4870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "0a8ed05e9c6b44fca9ff4b70f5c44333"
format_stamp: "Formatted at 2026-08-10 22:28:51 on dist-test-slave-njxd"
server_key: "cfad59d5ea2db1adf70dabf7202860a0"
server_key_iv: "cfb7a58a8b7cb42f8b89754fb060ef83"
server_key_version: "encryptionkey@0"
I20260810 22:28:51.829174 4870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:51.838044 4870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:51.841200 4870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:51.841403 4870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:51.842181 4870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:51.843017 4870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:51.843088 4870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:51.843155 4870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:51.843204 4870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:51.852061 4870 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:43509
I20260810 22:28:51.852128 5019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:43509 every 8 connection(s)
I20260810 22:28:51.853093 4870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:28:51.856176 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 4870
I20260810 22:28:51.856289 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:28:51.856621 1910 external_mini_cluster.cc:1428] Setting key e58773ffc0079b87dd2781dd0a024a8a
Aug 10 22:28:51 dist-test-slave-njxd krb5kdc[4592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400931, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260810 22:28:51.868515 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:51.855032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:59637 (local address 127.1.221.190:37457)
0810 22:28:51.855185 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:51.855189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:51.856149 (+ 960us) server_negotiation.cc:407] Connection header received
0810 22:28:51.857942 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:51.857945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:51.857999 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:51.858103 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:51.859851 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:51.860447 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:51.861700 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:51.861938 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:51.862061 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:51.864721 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:51.864725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:51.864728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:51.864763 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:51.866326 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:51.866869 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:51.866873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:51.866875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:51.866962 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:51.867317 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:51.867322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:51.867324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:51.867514 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:51.867628 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:51.868216 (+ 588us) server_negotiation.cc:299] Negotiation successful
0810 22:28:51.868341 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:28:51.869498 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:51.855446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37457 (local address 127.1.221.130:59637)
0810 22:28:51.855952 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:51.856014 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:51.857729 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:51.858332 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:51.858342 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:51.858920 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:51.859681 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:51.859706 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:51.860603 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:51.860606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:51.861506 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:51.861515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:51.862105 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:51.862109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:51.862304 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:51.862955 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:51.862972 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:51.864538 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:51.866456 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:51.866462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:51.866479 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:51.866761 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:51.867060 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:51.867064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:51.867066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:51.867217 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:51.867635 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:51.867644 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:51.867907 (+ 263us) client_negotiation.cc:769] Sending connection context
0810 22:28:51.868151 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 22:28:51.868499 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260810 22:28:51.870805 5020 heartbeater.cc:347] Connected to a master server at 127.1.221.190:37457
I20260810 22:28:51.871085 5020 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:51.871716 5020 heartbeater.cc:511] Master 127.1.221.190:37457 requested a full tablet report, sending...
I20260810 22:28:51.872941 4643 ts_manager.cc:194] Registered new tserver with Master: 0a8ed05e9c6b44fca9ff4b70f5c44333 (127.1.221.130:43509)
I20260810 22:28:51.873543 4643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:59637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:28:51.886831 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37457
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:52.002588 5025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:52.002914 5025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:52.002995 5025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:52.007135 5025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:52.007208 5025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:52.007290 5025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:28:52.012630 5025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:37457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:52.013715 5025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:52.014726 5025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:52.024363 5033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:52.024358 5031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:52.024499 5025 server_base.cc:1034] running on GCE node
W20260810 22:28:52.024669 5030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:52.025075 5025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:52.025810 5025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:52.026970 5025 hybrid_clock.cc:648] HybridClock initialized: now 1786400932026937 us; error 45 us; skew 500 ppm
Aug 10 22:28:52 dist-test-slave-njxd krb5kdc[4592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400932, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:52.029713 5025 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:28:52.030535 5025 webserver.cc:460] Webserver started at http://127.1.221.131:37597/ using document root <none> and password file <none>
I20260810 22:28:52.031095 5025 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:52.031188 5025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:52.031422 5025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:52.033543 5025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "15c026ecefd24122b8178861c5e5f612"
format_stamp: "Formatted at 2026-08-10 22:28:52 on dist-test-slave-njxd"
server_key: "9f2863f8fd6377888b04dc2efa131726"
server_key_iv: "089a6b1e7ef6185e7dcc300bd317f540"
server_key_version: "encryptionkey@0"
I20260810 22:28:52.034212 5025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "15c026ecefd24122b8178861c5e5f612"
format_stamp: "Formatted at 2026-08-10 22:28:52 on dist-test-slave-njxd"
server_key: "9f2863f8fd6377888b04dc2efa131726"
server_key_iv: "089a6b1e7ef6185e7dcc300bd317f540"
server_key_version: "encryptionkey@0"
I20260810 22:28:52.037954 5025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:28:52.040388 5040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:52.041810 5025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:28:52.041941 5025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "15c026ecefd24122b8178861c5e5f612"
format_stamp: "Formatted at 2026-08-10 22:28:52 on dist-test-slave-njxd"
server_key: "9f2863f8fd6377888b04dc2efa131726"
server_key_iv: "089a6b1e7ef6185e7dcc300bd317f540"
server_key_version: "encryptionkey@0"
I20260810 22:28:52.042049 5025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:52.053962 5025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:52.057204 5025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:52.057415 5025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:52.058087 5025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:52.058916 5025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:52.058966 5025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:52.059036 5025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:52.059079 5025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:52.067688 5025 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:44763
I20260810 22:28:52.067759 5166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:44763 every 8 connection(s)
I20260810 22:28:52.068686 5025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:28:52.074059 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 5025
I20260810 22:28:52.074209 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:28:52.074498 1910 external_mini_cluster.cc:1428] Setting key b50249d2d7495da2a12ef604d0393d0c
Aug 10 22:28:52 dist-test-slave-njxd krb5kdc[4592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400932, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:52.082916 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:52.070638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:59043 (local address 127.1.221.190:37457)
0810 22:28:52.070785 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:52.070789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:52.071654 (+ 865us) server_negotiation.cc:407] Connection header received
0810 22:28:52.073013 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:52.073016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:52.073076 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:52.073188 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:52.074527 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:52.075257 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:52.076426 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:52.076668 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:52.076790 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:52.079351 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:52.079355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:52.079357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:52.079394 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:52.080778 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:52.081291 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:52.081297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:52.081300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:52.081391 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:52.081877 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:52.081881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:52.081882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:52.082096 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:52.082191 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:52.082681 (+ 490us) server_negotiation.cc:299] Negotiation successful
0810 22:28:52.082794 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:28:52.083793 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:52.071008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:37457 (local address 127.1.221.131:59043)
0810 22:28:52.071510 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:52.071554 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:52.072797 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:52.073366 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:52.073373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:52.073810 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:52.074379 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:52.074396 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:52.075390 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:52.075393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:52.076297 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:52.076306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:52.076824 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:52.076828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:52.077073 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:52.077673 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:52.077690 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:52.079179 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:52.080905 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:52.080910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:52.080925 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:52.081182 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:52.081487 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:52.081491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:52.081493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:52.081715 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:52.082205 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:52.082210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:52.082454 (+ 244us) client_negotiation.cc:769] Sending connection context
0810 22:28:52.082607 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 22:28:52.082872 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":133,"threads_started":1}
I20260810 22:28:52.085004 5167 heartbeater.cc:347] Connected to a master server at 127.1.221.190:37457
I20260810 22:28:52.085258 5167 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:52.085834 5167 heartbeater.cc:511] Master 127.1.221.190:37457 requested a full tablet report, sending...
I20260810 22:28:52.086892 4643 ts_manager.cc:194] Registered new tserver with Master: 15c026ecefd24122b8178861c5e5f612 (127.1.221.131:44763)
I20260810 22:28:52.087561 4643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:59043
I20260810 22:28:52.088426 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 22:28:52.103729 4686 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:28:52.102289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.1.221.190:37457)
0810 22:28:52.102428 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:52.102432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:52.102643 (+ 211us) server_negotiation.cc:407] Connection header received
0810 22:28:52.102757 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:52.102760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:52.102803 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:52.102882 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:52.103546 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53062: BlockingRecv error: recv got EOF from 127.0.0.1:53062 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:28:52.104811 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 4713
I20260810 22:28:52.114593 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 4870
I20260810 22:28:52.122695 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 5025
I20260810 22:28:52.131150 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 4607
2026-08-10T22:28:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3829 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:28:52 dist-test-slave-njxd krb5kdc[5180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:28:52 dist-test-slave-njxd krb5kdc[5180](info): set up 2 sockets
Aug 10 22:28:52 dist-test-slave-njxd krb5kdc[5180](info): commencing operation
krb5kdc: starting...
W20260810 22:28:53.222748 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:28:53 dist-test-slave-njxd krb5kdc[5180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:28:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:28:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:28:53.350765 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:32843
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:32843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:53.468597 5195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:53.468871 5195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:53.468912 5195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:53.472983 5195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:28:53.473057 5195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:53.473078 5195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:53.473094 5195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:28:53.473109 5195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:28:53.478668 5195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:32843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:32843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:53.479808 5195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:53.480872 5195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:53.489619 5201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:53.489661 5205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:53.489660 5202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:53.489717 5195 server_base.cc:1034] running on GCE node
I20260810 22:28:53.490489 5195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:53.491613 5195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:53.492782 5195 hybrid_clock.cc:648] HybridClock initialized: now 1786400933492756 us; error 40 us; skew 500 ppm
Aug 10 22:28:53 dist-test-slave-njxd krb5kdc[5180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400933, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:53.495667 5195 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:28:53.496558 5195 webserver.cc:460] Webserver started at http://127.1.221.190:46037/ using document root <none> and password file <none>
I20260810 22:28:53.497120 5195 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:53.497182 5195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:53.497440 5195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:53.499540 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "03d936d76c1f46319dcadd8f06481a7c"
format_stamp: "Formatted at 2026-08-10 22:28:53 on dist-test-slave-njxd"
server_key: "92238cef3456808878d2afb21b6f1ed5"
server_key_iv: "9d105fa93667bd5781ae46b65133e0e6"
server_key_version: "encryptionkey@0"
I20260810 22:28:53.500185 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "03d936d76c1f46319dcadd8f06481a7c"
format_stamp: "Formatted at 2026-08-10 22:28:53 on dist-test-slave-njxd"
server_key: "92238cef3456808878d2afb21b6f1ed5"
server_key_iv: "9d105fa93667bd5781ae46b65133e0e6"
server_key_version: "encryptionkey@0"
I20260810 22:28:53.504014 5195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:28:53.506598 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:53.507825 5195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:28:53.507967 5195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "03d936d76c1f46319dcadd8f06481a7c"
format_stamp: "Formatted at 2026-08-10 22:28:53 on dist-test-slave-njxd"
server_key: "92238cef3456808878d2afb21b6f1ed5"
server_key_iv: "9d105fa93667bd5781ae46b65133e0e6"
server_key_version: "encryptionkey@0"
I20260810 22:28:53.508073 5195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:53.529405 5195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:53.532872 5195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:53.533093 5195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:53.540441 5195 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:32843
I20260810 22:28:53.540777 5274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:32843 every 8 connection(s)
I20260810 22:28:53.541492 5195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:28:53.544420 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:53.548471 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 5195
I20260810 22:28:53.548616 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:28:53.548897 1910 external_mini_cluster.cc:1428] Setting key b809a6c51e7caaa252f88598314534ff
I20260810 22:28:53.550436 5275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c: Bootstrap starting.
I20260810 22:28:53.553260 5275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:53.554224 5275 log.cc:824] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:53.556339 5275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c: No bootstrap required, opened a new log
Aug 10 22:28:53 dist-test-slave-njxd krb5kdc[5180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:53.560792 5275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d936d76c1f46319dcadd8f06481a7c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 32843 } }
I20260810 22:28:53.561012 5275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:53.561103 5275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d936d76c1f46319dcadd8f06481a7c, State: Initialized, Role: FOLLOWER
I20260810 22:28:53.561628 5275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d936d76c1f46319dcadd8f06481a7c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 32843 } }
I20260810 22:28:53.561805 5275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:28:53.561890 5275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:28:53.562005 5275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:53.563117 5275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d936d76c1f46319dcadd8f06481a7c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 32843 } }
I20260810 22:28:53.563494 5275 leader_election.cc:302] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03d936d76c1f46319dcadd8f06481a7c; no voters:
I20260810 22:28:53.563846 5275 leader_election.cc:288] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:28:53.563982 5280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:53.564199 5280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [term 1 LEADER]: Becoming Leader. State: Replica: 03d936d76c1f46319dcadd8f06481a7c, State: Running, Role: LEADER
I20260810 22:28:53.564949 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:53.550382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.1.221.190:32843)
0810 22:28:53.550898 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:53.550919 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:53.550961 (+ 42us) server_negotiation.cc:407] Connection header received
0810 22:28:53.551848 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:53.551875 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:53.552295 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:53.552670 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:53.553501 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:53.554933 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:53.555855 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:53.556462 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:53.556660 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:53.559449 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:53.559454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:53.559468 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:53.559495 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:53.561308 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:53.561966 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:53.561972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:53.561978 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:53.562139 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:53.562466 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:53.562467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:53.562469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:53.562841 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:53.562984 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:53.563357 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 22:28:53.563701 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":162,"threads_started":1}
I20260810 22:28:53.565245 5280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d936d76c1f46319dcadd8f06481a7c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 32843 } }
I20260810 22:28:53.565379 5275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:28:53.567080 5282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03d936d76c1f46319dcadd8f06481a7c. Latest consensus state: current_term: 1 leader_uuid: "03d936d76c1f46319dcadd8f06481a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d936d76c1f46319dcadd8f06481a7c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 32843 } } }
I20260810 22:28:53.567202 5282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:53.567579 5297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:28:53.567963 5281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03d936d76c1f46319dcadd8f06481a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d936d76c1f46319dcadd8f06481a7c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 32843 } } }
I20260810 22:28:53.568135 5281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:53.568365 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:28:53.570798 5297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:28:53.577409 5297 catalog_manager.cc:1347] Generated new cluster ID: 1f46b0213a6746db9ef45bfccf9a743c
I20260810 22:28:53.577540 5297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:28:53.587873 5297 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:28:53.588975 5297 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:28:53.590979 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:53.585007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59074 (local address 127.1.221.190:32843)
0810 22:28:53.585130 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:53.585135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:53.585430 (+ 295us) server_negotiation.cc:407] Connection header received
0810 22:28:53.585593 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:53.585597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:53.585661 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:53.585845 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:53.586530 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:53.587215 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:53.588460 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:53.588794 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:53.588905 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:53.589906 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:53.589910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:53.589913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:28:53.590479 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:53.590512 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:28:53.590531 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:53.590620 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:53.590755 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 22:28:53.590835 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50}
I20260810 22:28:53.600378 5297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03d936d76c1f46319dcadd8f06481a7c: Generated new TSK 0
I20260810 22:28:53.601125 5297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:28:53.613286 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 5195
2026-08-10T22:28:53Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1479 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:28:53 dist-test-slave-njxd krb5kdc[5311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:28:53 dist-test-slave-njxd krb5kdc[5311](info): set up 2 sockets
Aug 10 22:28:53 dist-test-slave-njxd krb5kdc[5311](info): commencing operation
krb5kdc: starting...
W20260810 22:28:54.698122 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:28:54 dist-test-slave-njxd krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:28:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:28:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:28:54.825241 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:36243
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:36243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:54.937333 5324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:54.937638 5324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:54.937682 5324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:54.941958 5324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:28:54.942020 5324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:54.942046 5324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:54.942152 5324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:28:54.942196 5324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:28:54.947512 5324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:36243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:36243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:54.948568 5324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:54.949646 5324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:54.958165 5334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:54.958165 5332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:54.958305 5324 server_base.cc:1034] running on GCE node
W20260810 22:28:54.958601 5329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:54.959218 5324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:54.960479 5324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:54.961675 5324 hybrid_clock.cc:648] HybridClock initialized: now 1786400934961651 us; error 34 us; skew 500 ppm
Aug 10 22:28:54 dist-test-slave-njxd krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400934, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:54.964532 5324 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:28:54.965443 5324 webserver.cc:460] Webserver started at http://127.1.221.190:39623/ using document root <none> and password file <none>
I20260810 22:28:54.966051 5324 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:54.966118 5324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:54.966286 5324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:54.968439 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "b1e3c15e521642fb9ff43f17636a1817"
format_stamp: "Formatted at 2026-08-10 22:28:54 on dist-test-slave-njxd"
server_key: "aedf9e856ea8ee22ecc9fcf59204f427"
server_key_iv: "91caabcfcee1e6d3e44b4f4ccc583b50"
server_key_version: "encryptionkey@0"
I20260810 22:28:54.969072 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "b1e3c15e521642fb9ff43f17636a1817"
format_stamp: "Formatted at 2026-08-10 22:28:54 on dist-test-slave-njxd"
server_key: "aedf9e856ea8ee22ecc9fcf59204f427"
server_key_iv: "91caabcfcee1e6d3e44b4f4ccc583b50"
server_key_version: "encryptionkey@0"
I20260810 22:28:54.972991 5324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:28:54.975473 5342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:54.976725 5324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:28:54.976856 5324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "b1e3c15e521642fb9ff43f17636a1817"
format_stamp: "Formatted at 2026-08-10 22:28:54 on dist-test-slave-njxd"
server_key: "aedf9e856ea8ee22ecc9fcf59204f427"
server_key_iv: "91caabcfcee1e6d3e44b4f4ccc583b50"
server_key_version: "encryptionkey@0"
I20260810 22:28:54.976960 5324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:55.000551 5324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:55.004132 5324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:55.004298 5324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:55.011492 5324 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:36243
I20260810 22:28:55.011787 5403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:36243 every 8 connection(s)
I20260810 22:28:55.012539 5324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:28:55.015398 5404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:55.021036 5404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817: Bootstrap starting.
I20260810 22:28:55.022040 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 5324
I20260810 22:28:55.022156 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:28:55.022464 1910 external_mini_cluster.cc:1428] Setting key 84f5b4af4482c408c6e3d6dfb82ede0d
I20260810 22:28:55.023962 5404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:55.024724 5404 log.cc:824] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:55.027037 5404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817: No bootstrap required, opened a new log
Aug 10 22:28:55 dist-test-slave-njxd krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:55.031615 5404 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e3c15e521642fb9ff43f17636a1817" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36243 } }
I20260810 22:28:55.031891 5404 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:55.031976 5404 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e3c15e521642fb9ff43f17636a1817, State: Initialized, Role: FOLLOWER
I20260810 22:28:55.032474 5404 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e3c15e521642fb9ff43f17636a1817" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36243 } }
I20260810 22:28:55.032720 5404 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:28:55.032861 5404 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:28:55.032982 5404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:55.033991 5404 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e3c15e521642fb9ff43f17636a1817" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36243 } }
I20260810 22:28:55.034341 5404 leader_election.cc:302] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1e3c15e521642fb9ff43f17636a1817; no voters:
I20260810 22:28:55.034698 5404 leader_election.cc:288] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:28:55.034777 5409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:55.034991 5409 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [term 1 LEADER]: Becoming Leader. State: Replica: b1e3c15e521642fb9ff43f17636a1817, State: Running, Role: LEADER
I20260810 22:28:55.035331 5409 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e3c15e521642fb9ff43f17636a1817" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36243 } }
I20260810 22:28:55.036048 5404 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:28:55.037322 5411 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1e3c15e521642fb9ff43f17636a1817" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e3c15e521642fb9ff43f17636a1817" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36243 } } }
I20260810 22:28:55.037467 5411 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:55.037622 5410 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1e3c15e521642fb9ff43f17636a1817. Latest consensus state: current_term: 1 leader_uuid: "b1e3c15e521642fb9ff43f17636a1817" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e3c15e521642fb9ff43f17636a1817" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36243 } } }
I20260810 22:28:55.037803 5410 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817 [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:55.037835 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:55.023600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.1.221.190:36243)
0810 22:28:55.024033 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:55.024049 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:55.024116 (+ 67us) server_negotiation.cc:407] Connection header received
0810 22:28:55.024786 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:55.024803 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:55.025255 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:55.025694 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:55.026559 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.027636 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.028848 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.029311 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.029429 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:55.032382 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:55.032388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:55.032405 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:55.032436 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:55.034349 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:55.034893 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:55.034899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:55.034905 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:55.035079 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:55.035416 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:55.035419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:55.035420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:55.035895 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:55.036128 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:55.036414 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 22:28:55.036752 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":142,"threads_started":1}
W20260810 22:28:55.041116 5426 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:28:55.041183 5426 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:28:55.041275 5419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:28:55.041999 5419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:28:55.046883 5419 catalog_manager.cc:1347] Generated new cluster ID: 918e3b43bfef4937a22491c26c4b14d3
I20260810 22:28:55.046950 5419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:28:55.064229 5419 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:28:55.065618 5419 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:28:55.075795 5419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1e3c15e521642fb9ff43f17636a1817: Generated new TSK 0
I20260810 22:28:55.076712 5419 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:28:55.116992 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36243
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:55.231807 5435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:55.232117 5435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:55.232198 5435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:55.236506 5435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:55.236588 5435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:55.236676 5435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:28:55.242316 5435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:55.243422 5435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:55.244491 5435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:55.254405 5442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:55.254405 5441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:55.254755 5435 server_base.cc:1034] running on GCE node
W20260810 22:28:55.254855 5444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:55.255393 5435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:55.256081 5435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:55.257284 5435 hybrid_clock.cc:648] HybridClock initialized: now 1786400935257249 us; error 47 us; skew 500 ppm
Aug 10 22:28:55 dist-test-slave-njxd krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400935, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:55.260248 5435 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:28:55.261219 5435 webserver.cc:460] Webserver started at http://127.1.221.129:33143/ using document root <none> and password file <none>
I20260810 22:28:55.261888 5435 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:55.261984 5435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:55.262230 5435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:55.264369 5435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "10802b6be0124f66927686ebf74f311f"
format_stamp: "Formatted at 2026-08-10 22:28:55 on dist-test-slave-njxd"
server_key: "8221e8023d11fd4624a05395461fdd4b"
server_key_iv: "3dc7d278153afbe0af93428e226abcf0"
server_key_version: "encryptionkey@0"
I20260810 22:28:55.264968 5435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "10802b6be0124f66927686ebf74f311f"
format_stamp: "Formatted at 2026-08-10 22:28:55 on dist-test-slave-njxd"
server_key: "8221e8023d11fd4624a05395461fdd4b"
server_key_iv: "3dc7d278153afbe0af93428e226abcf0"
server_key_version: "encryptionkey@0"
I20260810 22:28:55.269042 5435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:28:55.271780 5453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:55.273123 5435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:28:55.273264 5435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "10802b6be0124f66927686ebf74f311f"
format_stamp: "Formatted at 2026-08-10 22:28:55 on dist-test-slave-njxd"
server_key: "8221e8023d11fd4624a05395461fdd4b"
server_key_iv: "3dc7d278153afbe0af93428e226abcf0"
server_key_version: "encryptionkey@0"
I20260810 22:28:55.273389 5435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:55.290460 5435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:55.293692 5435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:55.293910 5435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:55.294616 5435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:55.295526 5435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:55.295579 5435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:55.295655 5435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:55.295684 5435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:55.305725 5435 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:44819
I20260810 22:28:55.305745 5585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:44819 every 8 connection(s)
I20260810 22:28:55.306713 5435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:28:55.314716 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 5435
I20260810 22:28:55.314873 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:28:55.315192 1910 external_mini_cluster.cc:1428] Setting key a80bc228173bd76c0e8a79bf6c35f761
Aug 10 22:28:55 dist-test-slave-njxd krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400935, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:55.321810 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:55.308657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:43399 (local address 127.1.221.190:36243)
0810 22:28:55.308813 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:55.308816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:55.309760 (+ 944us) server_negotiation.cc:407] Connection header received
0810 22:28:55.311248 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:55.311252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:55.311310 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:55.311388 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:55.312633 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.313141 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.314450 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.314824 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.314949 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:55.317848 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:55.317854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:55.317860 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:55.317897 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:55.319445 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:55.320108 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:55.320111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:55.320112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:55.320191 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:55.320601 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:55.320603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:55.320604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:55.320778 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:55.320852 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:55.321519 (+ 667us) server_negotiation.cc:299] Negotiation successful
0810 22:28:55.321681 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:28:55.323194 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:55.309057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36243 (local address 127.1.221.129:43399)
0810 22:28:55.309529 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:55.309576 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:55.311036 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:55.311570 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:55.311577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:55.311987 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:55.312504 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:55.312518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.313271 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.313273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:55.314315 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:55.314325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.315041 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.315045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:55.315194 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:55.315939 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:55.315960 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:55.317678 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:55.319574 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:55.319581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:55.319597 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:55.319994 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:55.320281 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:55.320285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:55.320287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:55.320496 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:55.320886 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:55.320895 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:55.321252 (+ 357us) client_negotiation.cc:769] Sending connection context
0810 22:28:55.321471 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 22:28:55.321934 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
I20260810 22:28:55.324579 5586 heartbeater.cc:347] Connected to a master server at 127.1.221.190:36243
I20260810 22:28:55.324843 5586 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:55.325371 5586 heartbeater.cc:511] Master 127.1.221.190:36243 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260810 22:28:55.327172 5364 ts_manager.cc:194] Registered new tserver with Master: 10802b6be0124f66927686ebf74f311f (127.1.221.129:44819)
I20260810 22:28:55.328333 5364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:43399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:28:55.348141 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36243
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:55.465260 5594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:55.465515 5594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:55.465557 5594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:55.469506 5594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:55.469568 5594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:55.469718 5594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:28:55.475200 5594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:55.476337 5594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:55.477458 5594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:55.487505 5599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:55.487545 5594 server_base.cc:1034] running on GCE node
W20260810 22:28:55.487505 5603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:55.487505 5601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:55.488232 5594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:55.488850 5594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:55.490015 5594 hybrid_clock.cc:648] HybridClock initialized: now 1786400935489990 us; error 36 us; skew 500 ppm
Aug 10 22:28:55 dist-test-slave-njxd krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400935, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:55.492851 5594 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:28:55.493816 5594 webserver.cc:460] Webserver started at http://127.1.221.130:41173/ using document root <none> and password file <none>
I20260810 22:28:55.494424 5594 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:55.494490 5594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:55.494752 5594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:55.496914 5594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "365dca65e7224876b803db1350f8cee9"
format_stamp: "Formatted at 2026-08-10 22:28:55 on dist-test-slave-njxd"
server_key: "ac0dd5e880b808e135319944bda3238b"
server_key_iv: "7c0f1a59ce877e850ca8d05148d910cb"
server_key_version: "encryptionkey@0"
I20260810 22:28:55.497514 5594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "365dca65e7224876b803db1350f8cee9"
format_stamp: "Formatted at 2026-08-10 22:28:55 on dist-test-slave-njxd"
server_key: "ac0dd5e880b808e135319944bda3238b"
server_key_iv: "7c0f1a59ce877e850ca8d05148d910cb"
server_key_version: "encryptionkey@0"
I20260810 22:28:55.501708 5594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:28:55.504254 5613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:55.505545 5594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:28:55.505751 5594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "365dca65e7224876b803db1350f8cee9"
format_stamp: "Formatted at 2026-08-10 22:28:55 on dist-test-slave-njxd"
server_key: "ac0dd5e880b808e135319944bda3238b"
server_key_iv: "7c0f1a59ce877e850ca8d05148d910cb"
server_key_version: "encryptionkey@0"
I20260810 22:28:55.505872 5594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:55.521379 5594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:55.524588 5594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:55.524806 5594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:55.525431 5594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:55.526353 5594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:55.526427 5594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:55.526487 5594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:55.526538 5594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:55.535422 5594 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:32937
I20260810 22:28:55.535427 5745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:32937 every 8 connection(s)
I20260810 22:28:55.536440 5594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
Aug 10 22:28:55 dist-test-slave-njxd krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400935, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:55.546149 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 5594
I20260810 22:28:55.546314 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:28:55.546598 1910 external_mini_cluster.cc:1428] Setting key 8627ffc2aa9222cb1f1bb36e978909a1
I20260810 22:28:55.550524 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:55.538716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:39641 (local address 127.1.221.190:36243)
0810 22:28:55.538857 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:55.538861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:55.539749 (+ 888us) server_negotiation.cc:407] Connection header received
0810 22:28:55.541106 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:55.541108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:55.541156 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:55.541229 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:55.542454 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.542974 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.544048 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.544260 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.544333 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:55.546790 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:55.546793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:55.546795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:55.546819 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:55.548415 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:55.548936 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:55.548940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:55.548942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:55.549027 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:55.549397 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:55.549402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:55.549403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:55.549652 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:55.549769 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:55.550255 (+ 486us) server_negotiation.cc:299] Negotiation successful
0810 22:28:55.550369 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:28:55.551363 5748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:55.539134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36243 (local address 127.1.221.130:39641)
0810 22:28:55.539595 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:55.539639 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:55.540892 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:55.541395 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:55.541402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:55.541842 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:55.542324 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:55.542338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.543081 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.543083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:55.543948 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:55.543956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.544387 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.544390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:55.544526 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:55.545062 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:55.545079 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:55.546604 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:55.548534 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:55.548541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:55.548555 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:55.548829 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:55.549137 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:55.549140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:55.549142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:55.549299 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:55.549769 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:55.549777 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:55.550015 (+ 238us) client_negotiation.cc:769] Sending connection context
0810 22:28:55.550211 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 22:28:55.550536 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":124,"threads_started":1}
I20260810 22:28:55.552646 5746 heartbeater.cc:347] Connected to a master server at 127.1.221.190:36243
I20260810 22:28:55.552888 5746 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:55.553376 5746 heartbeater.cc:511] Master 127.1.221.190:36243 requested a full tablet report, sending...
I20260810 22:28:55.554499 5364 ts_manager.cc:194] Registered new tserver with Master: 365dca65e7224876b803db1350f8cee9 (127.1.221.130:32937)
I20260810 22:28:55.555119 5364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:39641
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:28:55.577310 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36243
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:55.694782 5751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:55.695020 5751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:55.695101 5751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:55.700806 5751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:55.700886 5751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:55.700966 5751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:28:55.706568 5751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:36243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:55.707607 5751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:55.708621 5751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:55.718468 5758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:55.718490 5757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:55.718468 5760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:55.718760 5751 server_base.cc:1034] running on GCE node
I20260810 22:28:55.719107 5751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:55.719749 5751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:55.720942 5751 hybrid_clock.cc:648] HybridClock initialized: now 1786400935720932 us; error 33 us; skew 500 ppm
Aug 10 22:28:55 dist-test-slave-njxd krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400935, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:55.723755 5751 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:28:55.724639 5751 webserver.cc:460] Webserver started at http://127.1.221.131:40631/ using document root <none> and password file <none>
I20260810 22:28:55.725198 5751 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:55.725289 5751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:55.725524 5751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:55.727587 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "7b6633ad689f4f3fb3cbdfd1aa79849a"
format_stamp: "Formatted at 2026-08-10 22:28:55 on dist-test-slave-njxd"
server_key: "5080afaae24c9e8ba8f9dfd0136cdbb6"
server_key_iv: "b8792f95da5fde63947371446443e01d"
server_key_version: "encryptionkey@0"
I20260810 22:28:55.728163 5751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "7b6633ad689f4f3fb3cbdfd1aa79849a"
format_stamp: "Formatted at 2026-08-10 22:28:55 on dist-test-slave-njxd"
server_key: "5080afaae24c9e8ba8f9dfd0136cdbb6"
server_key_iv: "b8792f95da5fde63947371446443e01d"
server_key_version: "encryptionkey@0"
I20260810 22:28:55.731936 5751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:28:55.734603 5770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:55.735900 5751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:28:55.736037 5751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "7b6633ad689f4f3fb3cbdfd1aa79849a"
format_stamp: "Formatted at 2026-08-10 22:28:55 on dist-test-slave-njxd"
server_key: "5080afaae24c9e8ba8f9dfd0136cdbb6"
server_key_iv: "b8792f95da5fde63947371446443e01d"
server_key_version: "encryptionkey@0"
I20260810 22:28:55.736156 5751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:55.750753 5751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:55.754236 5751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:55.754453 5751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:55.755123 5751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:55.756040 5751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:55.756111 5751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:55.756234 5751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:55.756287 5751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:55.765969 5751 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:41043
I20260810 22:28:55.765992 5899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:41043 every 8 connection(s)
I20260810 22:28:55.766924 5751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:28:55.775166 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 5751
I20260810 22:28:55.775307 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:28:55.775657 1910 external_mini_cluster.cc:1428] Setting key 7aaa8580c866b4a182d3f5fa3946f19c
Aug 10 22:28:55 dist-test-slave-njxd krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400935, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:55.781281 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:55.768866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:48569 (local address 127.1.221.190:36243)
0810 22:28:55.769022 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:55.769025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:55.769954 (+ 929us) server_negotiation.cc:407] Connection header received
0810 22:28:55.771471 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:55.771474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:55.771526 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:55.771603 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:55.772805 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.773330 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.774515 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.774740 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.774820 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:55.777976 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:55.777979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:55.777981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:55.778024 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:55.779295 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:55.779787 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:55.779789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:55.779790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:55.779866 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:55.780291 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:55.780293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:55.780294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:55.780465 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:55.780541 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:55.781002 (+ 461us) server_negotiation.cc:299] Negotiation successful
0810 22:28:55.781136 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:28:55.782209 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:55.769263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:36243 (local address 127.1.221.131:48569)
0810 22:28:55.769783 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:55.769832 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:55.771229 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:55.771788 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:55.771795 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:55.772180 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:55.772674 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:55.772688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.773444 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.773447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:55.774407 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:55.774416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.774880 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.774883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:55.775026 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:55.775684 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:55.775719 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:55.777748 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:55.779400 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:55.779404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:55.779416 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:55.779689 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:55.779976 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:55.779979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:55.779981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:55.780187 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:55.780555 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:55.780560 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:55.780780 (+ 220us) client_negotiation.cc:769] Sending connection context
0810 22:28:55.780938 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 22:28:55.781283 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":112,"threads_started":1}
I20260810 22:28:55.783375 5900 heartbeater.cc:347] Connected to a master server at 127.1.221.190:36243
I20260810 22:28:55.783639 5900 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:55.784163 5900 heartbeater.cc:511] Master 127.1.221.190:36243 requested a full tablet report, sending...
I20260810 22:28:55.785161 5364 ts_manager.cc:194] Registered new tserver with Master: 7b6633ad689f4f3fb3cbdfd1aa79849a (127.1.221.131:41043)
I20260810 22:28:55.785774 5364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:48569
I20260810 22:28:55.789772 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 22:28:55 dist-test-slave-njxd krb5kdc[5311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400935, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 22:28:55 dist-test-slave-njxd krb5kdc[5311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400935, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:55.816485 5407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:55.808297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45680 (local address 127.1.221.190:36243)
0810 22:28:55.808434 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:55.808437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:55.808576 (+ 139us) server_negotiation.cc:407] Connection header received
0810 22:28:55.808693 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:55.808696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:55.808741 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:55.808817 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:55.809660 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.810185 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.811018 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:55.811234 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:55.811314 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:55.813727 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:55.813731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:55.813733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:55.813758 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:55.815028 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:55.815416 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:55.815419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:55.815420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:55.815495 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:55.815724 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:55.815726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:55.815728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:55.815943 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:55.816045 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:55.816259 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 22:28:55.816359 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 22:28:55.817003 5364 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45680
I20260810 22:28:55.817902 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 5435
I20260810 22:28:55.825999 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 5594
I20260810 22:28:55.834052 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 5751
I20260810 22:28:55.846565 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 5324
2026-08-10T22:28:55Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2235 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:28:55 dist-test-slave-njxd krb5kdc[5915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:28:55 dist-test-slave-njxd krb5kdc[5915](info): set up 2 sockets
Aug 10 22:28:55 dist-test-slave-njxd krb5kdc[5915](info): commencing operation
krb5kdc: starting...
W20260810 22:28:56.947008 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:28:57 dist-test-slave-njxd krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:28:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:28:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:28:57.077127 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:45785
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:45785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:57.189618 5930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:57.189888 5930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:57.189926 5930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:57.193778 5930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:28:57.193825 5930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:57.193843 5930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:57.193859 5930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:28:57.193874 5930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:28:57.199011 5930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:45785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:45785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:57.200037 5930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:57.201030 5930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:57.209697 5938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:57.209703 5936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:57.209789 5930 server_base.cc:1034] running on GCE node
W20260810 22:28:57.209957 5935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:57.210510 5930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:57.211627 5930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:57.212818 5930 hybrid_clock.cc:648] HybridClock initialized: now 1786400937212801 us; error 36 us; skew 500 ppm
Aug 10 22:28:57 dist-test-slave-njxd krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400937, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:57.220535 5930 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:28:57.221455 5930 webserver.cc:460] Webserver started at http://127.1.221.190:37795/ using document root <none> and password file <none>
I20260810 22:28:57.222110 5930 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:57.222206 5930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:57.222456 5930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:57.224658 5930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "dcbd118497504f9fae139e70ca18209a"
format_stamp: "Formatted at 2026-08-10 22:28:57 on dist-test-slave-njxd"
server_key: "70a8ebd7e70c607cb34997ae5cb32210"
server_key_iv: "3bd8781eb050fa363ebca17042654532"
server_key_version: "encryptionkey@0"
I20260810 22:28:57.225258 5930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "dcbd118497504f9fae139e70ca18209a"
format_stamp: "Formatted at 2026-08-10 22:28:57 on dist-test-slave-njxd"
server_key: "70a8ebd7e70c607cb34997ae5cb32210"
server_key_iv: "3bd8781eb050fa363ebca17042654532"
server_key_version: "encryptionkey@0"
I20260810 22:28:57.229130 5930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 22:28:57.231666 5946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:57.232909 5930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:28:57.233059 5930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "dcbd118497504f9fae139e70ca18209a"
format_stamp: "Formatted at 2026-08-10 22:28:57 on dist-test-slave-njxd"
server_key: "70a8ebd7e70c607cb34997ae5cb32210"
server_key_iv: "3bd8781eb050fa363ebca17042654532"
server_key_version: "encryptionkey@0"
I20260810 22:28:57.233179 5930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:57.251358 5930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:57.254439 5930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:57.254652 5930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:57.261910 5930 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:45785
I20260810 22:28:57.261940 6004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:45785 every 8 connection(s)
I20260810 22:28:57.262876 5930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:28:57.264833 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 5930
I20260810 22:28:57.264943 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:28:57.265236 1910 external_mini_cluster.cc:1428] Setting key 5a82c1fdcd264a569963bd847699083a
I20260810 22:28:57.265975 6005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:57.273226 6005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a: Bootstrap starting.
Aug 10 22:28:57 dist-test-slave-njxd krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:57.276022 6005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:57.276822 6005 log.cc:824] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:57.279183 6005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a: No bootstrap required, opened a new log
I20260810 22:28:57.281361 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:57.266678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.1.221.190:45785)
0810 22:28:57.267136 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:57.267160 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:57.267205 (+ 45us) server_negotiation.cc:407] Connection header received
0810 22:28:57.268115 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:57.268140 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:57.268564 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:57.268952 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:57.269667 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:57.271222 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:57.272097 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:57.272807 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:57.272973 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:57.276030 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:57.276035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:57.276049 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:57.276106 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:57.278228 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:57.278681 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:57.278687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:57.278693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:57.278838 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:57.279093 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:57.279096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:57.279097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:57.279623 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:57.279832 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:57.280096 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 22:28:57.280400 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":147,"threads_started":1}
I20260810 22:28:57.283747 6005 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbd118497504f9fae139e70ca18209a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45785 } }
I20260810 22:28:57.283958 6005 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:57.283993 6005 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbd118497504f9fae139e70ca18209a, State: Initialized, Role: FOLLOWER
I20260810 22:28:57.284341 6005 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbd118497504f9fae139e70ca18209a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45785 } }
I20260810 22:28:57.284444 6005 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:28:57.284480 6005 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:28:57.284547 6005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:28:57.285425 6005 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbd118497504f9fae139e70ca18209a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45785 } }
I20260810 22:28:57.285782 6005 leader_election.cc:302] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcbd118497504f9fae139e70ca18209a; no voters:
I20260810 22:28:57.286036 6005 leader_election.cc:288] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:28:57.286226 6012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:57.286473 6012 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [term 1 LEADER]: Becoming Leader. State: Replica: dcbd118497504f9fae139e70ca18209a, State: Running, Role: LEADER
I20260810 22:28:57.286763 6012 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbd118497504f9fae139e70ca18209a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45785 } }
I20260810 22:28:57.287230 6005 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:28:57.288584 6013 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcbd118497504f9fae139e70ca18209a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbd118497504f9fae139e70ca18209a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45785 } } }
I20260810 22:28:57.288527 6014 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcbd118497504f9fae139e70ca18209a. Latest consensus state: current_term: 1 leader_uuid: "dcbd118497504f9fae139e70ca18209a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbd118497504f9fae139e70ca18209a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45785 } } }
I20260810 22:28:57.288692 6013 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:57.288692 6014 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:57.289966 6021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:28:57.292975 6021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:28:57.299008 6021 catalog_manager.cc:1347] Generated new cluster ID: ee3793ab01f4451cbb9e5ce3e6aed7a3
I20260810 22:28:57.299103 6021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:28:57.308740 6021 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:28:57.309698 6021 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:28:57.316298 6021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcbd118497504f9fae139e70ca18209a: Generated new TSK 0
I20260810 22:28:57.317000 6021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:28:57.354040 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45785
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:57.485886 6033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:57.486159 6033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:57.486238 6033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:57.490489 6033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:57.490559 6033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:57.490649 6033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:28:57.496145 6033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:57.497323 6033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:57.498494 6033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:57.508944 6039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:57.508971 6040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:57.509133 6033 server_base.cc:1034] running on GCE node
W20260810 22:28:57.508944 6042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:57.509756 6033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:57.510393 6033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:57.511565 6033 hybrid_clock.cc:648] HybridClock initialized: now 1786400937511540 us; error 37 us; skew 500 ppm
Aug 10 22:28:57 dist-test-slave-njxd krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400937, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:57.514372 6033 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:28:57.515252 6033 webserver.cc:460] Webserver started at http://127.1.221.129:35431/ using document root <none> and password file <none>
I20260810 22:28:57.515823 6033 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:57.515885 6033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:57.516053 6033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:57.518174 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "5a3dcd8314674c21a82e12e241915586"
format_stamp: "Formatted at 2026-08-10 22:28:57 on dist-test-slave-njxd"
server_key: "d7cd845ac6efc0d1804aa54d5365bcef"
server_key_iv: "7f7b5a2ee6beddeba99a0da8587ccbf7"
server_key_version: "encryptionkey@0"
I20260810 22:28:57.518687 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "5a3dcd8314674c21a82e12e241915586"
format_stamp: "Formatted at 2026-08-10 22:28:57 on dist-test-slave-njxd"
server_key: "d7cd845ac6efc0d1804aa54d5365bcef"
server_key_iv: "7f7b5a2ee6beddeba99a0da8587ccbf7"
server_key_version: "encryptionkey@0"
I20260810 22:28:57.522408 6033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:28:57.524835 6049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:57.526190 6033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:28:57.526293 6033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "5a3dcd8314674c21a82e12e241915586"
format_stamp: "Formatted at 2026-08-10 22:28:57 on dist-test-slave-njxd"
server_key: "d7cd845ac6efc0d1804aa54d5365bcef"
server_key_iv: "7f7b5a2ee6beddeba99a0da8587ccbf7"
server_key_version: "encryptionkey@0"
I20260810 22:28:57.526383 6033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:57.561349 6033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:57.564829 6033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:57.564996 6033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:57.565630 6033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:57.566533 6033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:57.566583 6033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:57.566635 6033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:57.566658 6033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:57.576684 6033 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:34443
I20260810 22:28:57.576706 6181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:34443 every 8 connection(s)
I20260810 22:28:57.577793 6033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:28:57.582865 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 6033
I20260810 22:28:57.582985 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:28:57.583297 1910 external_mini_cluster.cc:1428] Setting key fde7ae70ecc5eafbaa608f67794f96c5
Aug 10 22:28:57 dist-test-slave-njxd krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400937, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:57.593340 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:57.579699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:60517 (local address 127.1.221.190:45785)
0810 22:28:57.579866 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:57.579871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:57.580880 (+ 1009us) server_negotiation.cc:407] Connection header received
0810 22:28:57.582367 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:57.582370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:57.582423 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:57.582500 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:57.583950 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:57.584502 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:57.585763 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:57.586009 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:57.586125 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:57.589301 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:57.589308 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:57.589313 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:57.589347 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:57.590949 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:57.591661 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:57.591664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:57.591665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:57.591740 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:57.592178 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:57.592180 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:57.592182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:57.592359 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:57.592438 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:57.593095 (+ 657us) server_negotiation.cc:299] Negotiation successful
0810 22:28:57.593203 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:28:57.594682 6184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:57.580192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:45785 (local address 127.1.221.129:60517)
0810 22:28:57.580683 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:57.580730 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:57.582142 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:57.582679 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:57.582687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:57.583196 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:57.583766 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:57.583781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:57.584645 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:57.584648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:57.585603 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:57.585638 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:57.586174 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:57.586178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:57.586362 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:57.587016 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:57.587034 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:57.589065 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:57.591081 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:57.591091 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:57.591120 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:57.591551 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:57.591841 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:57.591846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:57.591850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:57.592063 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:57.592477 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:57.592485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:57.592827 (+ 342us) client_negotiation.cc:769] Sending connection context
0810 22:28:57.593064 (+ 237us) client_negotiation.cc:240] Negotiation successful
0810 22:28:57.593452 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260810 22:28:57.596068 6182 heartbeater.cc:347] Connected to a master server at 127.1.221.190:45785
I20260810 22:28:57.596352 6182 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:57.596879 6182 heartbeater.cc:511] Master 127.1.221.190:45785 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260810 22:28:57.598640 5965 ts_manager.cc:194] Registered new tserver with Master: 5a3dcd8314674c21a82e12e241915586 (127.1.221.129:34443)
I20260810 22:28:57.599788 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:60517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:28:57.614856 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45785
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:57.729826 6187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:57.730119 6187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:57.730190 6187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:57.734200 6187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:57.734300 6187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:57.734390 6187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:28:57.739688 6187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:57.740880 6187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:57.741955 6187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:57.751544 6192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:57.751592 6193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:57.751634 6187 server_base.cc:1034] running on GCE node
W20260810 22:28:57.751763 6195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:57.752254 6187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:57.752893 6187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:57.754079 6187 hybrid_clock.cc:648] HybridClock initialized: now 1786400937754049 us; error 50 us; skew 500 ppm
Aug 10 22:28:57 dist-test-slave-njxd krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400937, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:57.756997 6187 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:28:57.757985 6187 webserver.cc:460] Webserver started at http://127.1.221.130:44381/ using document root <none> and password file <none>
I20260810 22:28:57.758509 6187 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:57.758569 6187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:57.758749 6187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:57.760820 6187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "280f415e456648cbb04d8d349c7ce083"
format_stamp: "Formatted at 2026-08-10 22:28:57 on dist-test-slave-njxd"
server_key: "4291df668c9ba76ff2a76e0d9f862a34"
server_key_iv: "c31a25f3172080f7288ec075557c53a5"
server_key_version: "encryptionkey@0"
I20260810 22:28:57.761451 6187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "280f415e456648cbb04d8d349c7ce083"
format_stamp: "Formatted at 2026-08-10 22:28:57 on dist-test-slave-njxd"
server_key: "4291df668c9ba76ff2a76e0d9f862a34"
server_key_iv: "c31a25f3172080f7288ec075557c53a5"
server_key_version: "encryptionkey@0"
I20260810 22:28:57.765326 6187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 22:28:57.767915 6204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:57.769153 6187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:28:57.769286 6187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "280f415e456648cbb04d8d349c7ce083"
format_stamp: "Formatted at 2026-08-10 22:28:57 on dist-test-slave-njxd"
server_key: "4291df668c9ba76ff2a76e0d9f862a34"
server_key_iv: "c31a25f3172080f7288ec075557c53a5"
server_key_version: "encryptionkey@0"
I20260810 22:28:57.769392 6187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:57.781780 6187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:57.784641 6187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:57.784843 6187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:57.785446 6187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:57.786291 6187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:57.786338 6187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:57.786404 6187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:57.786441 6187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:57.795845 6187 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:38887
I20260810 22:28:57.795866 6336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:38887 every 8 connection(s)
I20260810 22:28:57.796813 6187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:28:57.801945 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 6187
I20260810 22:28:57.802075 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:28:57.802418 1910 external_mini_cluster.cc:1428] Setting key 68bbf54ca6b18d45d88d4427b5ac001e
Aug 10 22:28:57 dist-test-slave-njxd krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400937, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:57.812892 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:57.798919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:39763 (local address 127.1.221.190:45785)
0810 22:28:57.799080 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:57.799085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:57.800060 (+ 975us) server_negotiation.cc:407] Connection header received
0810 22:28:57.801566 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:57.801569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:57.801719 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:57.801801 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:57.803349 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:57.804065 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:57.805899 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:57.806146 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:57.806260 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:57.808814 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:57.808821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:57.808827 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:57.808863 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:57.810688 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:57.811199 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:57.811206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:57.811211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:57.811310 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:57.811660 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:57.811665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:57.811667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:57.811914 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:57.812036 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:57.812552 (+ 516us) server_negotiation.cc:299] Negotiation successful
0810 22:28:57.812698 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:28:57.813740 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:57.799388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:45785 (local address 127.1.221.130:39763)
0810 22:28:57.799892 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:57.799939 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:57.801254 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:57.802057 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:57.802069 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:57.802618 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:57.803210 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:57.803227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:57.804202 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:57.804206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:57.805780 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:57.805795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:57.806310 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:57.806315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:57.806487 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:57.807125 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:57.807143 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:57.808628 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:57.810813 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:57.810818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:57.810830 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:57.811091 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:57.811411 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:57.811413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:57.811414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:57.811550 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:57.812041 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:57.812046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:57.812296 (+ 250us) client_negotiation.cc:769] Sending connection context
0810 22:28:57.812457 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 22:28:57.812720 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260810 22:28:57.815105 6337 heartbeater.cc:347] Connected to a master server at 127.1.221.190:45785
I20260810 22:28:57.815369 6337 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:57.816005 6337 heartbeater.cc:511] Master 127.1.221.190:45785 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260810 22:28:57.817309 5965 ts_manager.cc:194] Registered new tserver with Master: 280f415e456648cbb04d8d349c7ce083 (127.1.221.130:38887)
I20260810 22:28:57.818202 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:39763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:28:57.832885 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45785
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:57.952636 6343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:57.952906 6343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:57.952951 6343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:57.957158 6343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:57.957223 6343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:57.957319 6343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:28:57.963624 6343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:57.964792 6343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:57.965958 6343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:28:57.976926 6343 server_base.cc:1034] running on GCE node
W20260810 22:28:57.976900 6349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:57.976948 6351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:57.976900 6348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:57.977707 6343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:57.978379 6343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:57.979544 6343 hybrid_clock.cc:648] HybridClock initialized: now 1786400937979527 us; error 31 us; skew 500 ppm
Aug 10 22:28:57 dist-test-slave-njxd krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400937, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:57.982383 6343 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:28:57.983233 6343 webserver.cc:460] Webserver started at http://127.1.221.131:41035/ using document root <none> and password file <none>
I20260810 22:28:57.983769 6343 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:57.983829 6343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:57.983995 6343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:57.986056 6343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "7ec71ca84b8f474fb2d22a8c038a599c"
format_stamp: "Formatted at 2026-08-10 22:28:57 on dist-test-slave-njxd"
server_key: "9a6ca62bf356027b90bed8483c6cb5c1"
server_key_iv: "c928c01f537dfe3233c7e5fe8fe9a1c0"
server_key_version: "encryptionkey@0"
I20260810 22:28:57.986562 6343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "7ec71ca84b8f474fb2d22a8c038a599c"
format_stamp: "Formatted at 2026-08-10 22:28:57 on dist-test-slave-njxd"
server_key: "9a6ca62bf356027b90bed8483c6cb5c1"
server_key_iv: "c928c01f537dfe3233c7e5fe8fe9a1c0"
server_key_version: "encryptionkey@0"
I20260810 22:28:57.990533 6343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:28:57.992954 6360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:57.994258 6343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:28:57.994401 6343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "7ec71ca84b8f474fb2d22a8c038a599c"
format_stamp: "Formatted at 2026-08-10 22:28:57 on dist-test-slave-njxd"
server_key: "9a6ca62bf356027b90bed8483c6cb5c1"
server_key_iv: "c928c01f537dfe3233c7e5fe8fe9a1c0"
server_key_version: "encryptionkey@0"
I20260810 22:28:57.994519 6343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:58.011528 6343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:58.014938 6343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:58.015146 6343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:58.015759 6343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:58.016585 6343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:58.016656 6343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:58.016728 6343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:58.016788 6343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:58.026405 6343 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:43089
I20260810 22:28:58.026434 6487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:43089 every 8 connection(s)
I20260810 22:28:58.027321 6343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:28:58.030632 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 6343
I20260810 22:28:58.030786 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:28:58.031191 1910 external_mini_cluster.cc:1428] Setting key b0468c01d97c2851ba94f26216469feb
Aug 10 22:28:58 dist-test-slave-njxd krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400937, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:58.042024 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:58.029144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:51579 (local address 127.1.221.190:45785)
0810 22:28:58.029288 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:58.029291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:58.030330 (+ 1039us) server_negotiation.cc:407] Connection header received
0810 22:28:58.032207 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:58.032212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:58.032279 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:58.032357 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:58.033885 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:58.034427 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:58.035617 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:58.035851 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:58.035959 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:58.038596 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:58.038599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:58.038601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:58.038647 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:58.040062 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:58.040552 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:58.040554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:58.040556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:58.040657 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:58.040973 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:58.040975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:58.040976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:58.041157 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:58.041232 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:58.041789 (+ 557us) server_negotiation.cc:299] Negotiation successful
0810 22:28:58.041900 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:28:58.042858 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:58.029495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:45785 (local address 127.1.221.131:51579)
0810 22:28:58.030180 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:58.030223 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:58.031947 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:58.032573 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:58.032601 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:58.033028 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:58.033743 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:58.033760 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:58.034571 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:58.034574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:58.035463 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:58.035472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:58.036012 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:58.036016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:58.036209 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:58.036791 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:58.036808 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:58.038425 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:58.040173 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:58.040178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:58.040192 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:58.040455 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:58.040737 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:58.040739 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:58.040740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:58.040886 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:58.041243 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:58.041248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:58.041487 (+ 239us) client_negotiation.cc:769] Sending connection context
0810 22:28:58.041705 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 22:28:58.041987 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":126,"threads_started":1}
I20260810 22:28:58.044113 6488 heartbeater.cc:347] Connected to a master server at 127.1.221.190:45785
I20260810 22:28:58.044396 6488 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:58.044935 6488 heartbeater.cc:511] Master 127.1.221.190:45785 requested a full tablet report, sending...
I20260810 22:28:58.045954 5965 ts_manager.cc:194] Registered new tserver with Master: 7ec71ca84b8f474fb2d22a8c038a599c (127.1.221.131:43089)
I20260810 22:28:58.046487 5965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:51579
I20260810 22:28:58.057163 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 22:28:58 dist-test-slave-njxd krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:28:58 dist-test-slave-njxd krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260810 22:28:58.086431 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:58.076270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.1.221.129:34443)
0810 22:28:58.076614 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:58.076631 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:58.076664 (+ 33us) server_negotiation.cc:407] Connection header received
0810 22:28:58.076719 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:58.076724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:58.076989 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:58.077144 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:58.077938 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:58.078737 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:58.079688 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:58.080097 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:58.080184 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:58.082703 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:58.082715 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:58.082731 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:58.082759 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:58.084552 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:58.085002 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:58.085008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:58.085009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:58.085096 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:58.085342 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:58.085344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:58.085345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:58.085680 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:58.085882 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:58.086020 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 22:28:58.086180 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":126,"threads_started":1}
Aug 10 22:28:58 dist-test-slave-njxd krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:58.098554 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:58.089823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.1.221.190:45785)
0810 22:28:58.089975 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:58.089979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:58.090090 (+ 111us) server_negotiation.cc:407] Connection header received
0810 22:28:58.090192 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:58.090194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:58.090236 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:58.090311 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:58.090938 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:58.091457 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:58.092233 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:58.092469 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:58.092584 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:58.095178 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:58.095182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:58.095184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:58.095223 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:58.096723 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:58.097168 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:58.097172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:58.097174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:58.097264 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:58.097565 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:58.097569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:58.097570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:58.097846 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:58.097999 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:58.098243 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 22:28:58.098384 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 22:28:58.099021 5965 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59968
I20260810 22:28:58.099750 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 6033
I20260810 22:28:58.113343 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 6187
I20260810 22:28:58.123543 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 6343
I20260810 22:28:58.131132 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 5930
2026-08-10T22:28:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2283 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:28:58 dist-test-slave-njxd krb5kdc[6500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:28:58 dist-test-slave-njxd krb5kdc[6500](info): set up 2 sockets
Aug 10 22:28:58 dist-test-slave-njxd krb5kdc[6500](info): commencing operation
krb5kdc: starting...
W20260810 22:28:59.234428 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:28:59 dist-test-slave-njxd krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:28:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:28:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:28:59.364898 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42049
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:42049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:59.482354 6514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:59.482687 6514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:59.482774 6514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:59.486891 6514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:28:59.486992 6514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:59.487035 6514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:59.487097 6514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:28:59.487156 6514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:28:59.492547 6514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:42049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:59.493687 6514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:59.494786 6514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:59.503140 6520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:59.503141 6521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:59.503147 6524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:59.503408 6514 server_base.cc:1034] running on GCE node
I20260810 22:28:59.504228 6514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:59.505288 6514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:59.506475 6514 hybrid_clock.cc:648] HybridClock initialized: now 1786400939506432 us; error 56 us; skew 500 ppm
Aug 10 22:28:59 dist-test-slave-njxd krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400939, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:59.509183 6514 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:28:59.509984 6514 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:59.510052 6514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:59.510231 6514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:59.512320 6514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "4e32441cb65f4b22bc810f9df9a2b239"
format_stamp: "Formatted at 2026-08-10 22:28:59 on dist-test-slave-njxd"
server_key: "0b394bbf6cc681290ecd7633990816c1"
server_key_iv: "a606a55ea84062b6f7f000835c74ef02"
server_key_version: "encryptionkey@0"
I20260810 22:28:59.512840 6514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "4e32441cb65f4b22bc810f9df9a2b239"
format_stamp: "Formatted at 2026-08-10 22:28:59 on dist-test-slave-njxd"
server_key: "0b394bbf6cc681290ecd7633990816c1"
server_key_iv: "a606a55ea84062b6f7f000835c74ef02"
server_key_version: "encryptionkey@0"
I20260810 22:28:59.516569 6514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:28:59.518978 6531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:59.520272 6514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:28:59.520419 6514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "4e32441cb65f4b22bc810f9df9a2b239"
format_stamp: "Formatted at 2026-08-10 22:28:59 on dist-test-slave-njxd"
server_key: "0b394bbf6cc681290ecd7633990816c1"
server_key_iv: "a606a55ea84062b6f7f000835c74ef02"
server_key_version: "encryptionkey@0"
I20260810 22:28:59.520526 6514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:59.540473 6514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:59.543495 6514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:59.543730 6514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:59.551173 6514 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:42049
I20260810 22:28:59.551537 6590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:42049 every 8 connection(s)
I20260810 22:28:59.551909 6514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:28:59.554715 6592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:28:59.560165 6592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239: Bootstrap starting.
I20260810 22:28:59.562053 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 6514
I20260810 22:28:59.562202 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:28:59.562486 1910 external_mini_cluster.cc:1428] Setting key 2113619546ecab0324e75c19b3223ceb
I20260810 22:28:59.562716 6592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239: Neither blocks nor log segments found. Creating new log.
I20260810 22:28:59.563688 6592 log.cc:824] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239: Log is configured to *not* fsync() on all Append() calls
I20260810 22:28:59.565714 6592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239: No bootstrap required, opened a new log
I20260810 22:28:59.570130 6592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e32441cb65f4b22bc810f9df9a2b239" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42049 } }
I20260810 22:28:59.570427 6592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:28:59.570483 6592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e32441cb65f4b22bc810f9df9a2b239, State: Initialized, Role: FOLLOWER
I20260810 22:28:59.571033 6592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e32441cb65f4b22bc810f9df9a2b239" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42049 } }
I20260810 22:28:59.571193 6592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:28:59.571277 6592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:28:59.571396 6592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 22:28:59 dist-test-slave-njxd krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:59.572476 6592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e32441cb65f4b22bc810f9df9a2b239" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42049 } }
I20260810 22:28:59.572877 6592 leader_election.cc:302] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e32441cb65f4b22bc810f9df9a2b239; no voters:
I20260810 22:28:59.573232 6592 leader_election.cc:288] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:28:59.573400 6597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:28:59.573639 6597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [term 1 LEADER]: Becoming Leader. State: Replica: 4e32441cb65f4b22bc810f9df9a2b239, State: Running, Role: LEADER
I20260810 22:28:59.573978 6597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e32441cb65f4b22bc810f9df9a2b239" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42049 } }
I20260810 22:28:59.574683 6592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:28:59.577656 6600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e32441cb65f4b22bc810f9df9a2b239. Latest consensus state: current_term: 1 leader_uuid: "4e32441cb65f4b22bc810f9df9a2b239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e32441cb65f4b22bc810f9df9a2b239" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42049 } } }
I20260810 22:28:59.577862 6600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:59.577859 6598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e32441cb65f4b22bc810f9df9a2b239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e32441cb65f4b22bc810f9df9a2b239" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42049 } } }
I20260810 22:28:59.578083 6598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239 [sys.catalog]: This master's current role is: LEADER
I20260810 22:28:59.578608 6595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:59.563625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.1.221.190:42049)
0810 22:28:59.564127 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:59.564145 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:59.564181 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:28:59.564806 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:59.564822 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:59.565131 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:59.565400 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:59.566389 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:59.567384 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:59.568688 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:59.569388 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:59.569513 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:59.572697 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:59.572702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:59.572717 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:59.572746 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:59.574974 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:59.575511 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:59.575519 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:59.575526 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:59.575684 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:59.576053 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:59.576058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:59.576059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:59.576563 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:59.576783 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:59.577164 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 22:28:59.577521 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":170,"threads_started":1}
W20260810 22:28:59.580331 6613 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:28:59.580411 6613 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:28:59.580482 6614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:28:59.581312 6614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:28:59.587244 6614 catalog_manager.cc:1347] Generated new cluster ID: a00968abf24e4bd0848efd7f177392be
I20260810 22:28:59.587312 6614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:28:59.594722 6614 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:28:59.595911 6614 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:28:59.602813 6614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239: Generated new TSK 0
I20260810 22:28:59.603597 6614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:28:59.634985 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42049
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:59.767973 6619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:59.768275 6619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:59.768360 6619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:28:59.772538 6619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:28:59.772629 6619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:28:59.772724 6619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:28:59.778393 6619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:28:59.779546 6619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:28:59.780642 6619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:28:59.790218 6625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:59.790335 6619 server_base.cc:1034] running on GCE node
W20260810 22:28:59.790218 6626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:28:59.790218 6628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:28:59.791060 6619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:28:59.791718 6619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:28:59.792904 6619 hybrid_clock.cc:648] HybridClock initialized: now 1786400939792872 us; error 45 us; skew 500 ppm
Aug 10 22:28:59 dist-test-slave-njxd krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400939, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:28:59.795816 6619 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:28:59.796532 6619 fs_manager.cc:359] Metadata directory not provided
I20260810 22:28:59.796639 6619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:28:59.796881 6619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:28:59.799036 6619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "1951b1d2cae9451b915906e4c0a0ceb3"
format_stamp: "Formatted at 2026-08-10 22:28:59 on dist-test-slave-njxd"
server_key: "dd4b067ee62068fc21b9db5c8c5068c6"
server_key_iv: "d13538c2f363bc59144e42da43257586"
server_key_version: "encryptionkey@0"
I20260810 22:28:59.799640 6619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "1951b1d2cae9451b915906e4c0a0ceb3"
format_stamp: "Formatted at 2026-08-10 22:28:59 on dist-test-slave-njxd"
server_key: "dd4b067ee62068fc21b9db5c8c5068c6"
server_key_iv: "d13538c2f363bc59144e42da43257586"
server_key_version: "encryptionkey@0"
I20260810 22:28:59.803413 6619 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 22:28:59.805857 6635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:59.807120 6619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:28:59.807268 6619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "1951b1d2cae9451b915906e4c0a0ceb3"
format_stamp: "Formatted at 2026-08-10 22:28:59 on dist-test-slave-njxd"
server_key: "dd4b067ee62068fc21b9db5c8c5068c6"
server_key_iv: "d13538c2f363bc59144e42da43257586"
server_key_version: "encryptionkey@0"
I20260810 22:28:59.807391 6619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:28:59.829172 6619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:28:59.832188 6619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:28:59.832409 6619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:28:59.833046 6619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:28:59.833947 6619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:28:59.834021 6619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:59.834101 6619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:28:59.834146 6619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:28:59.843667 6619 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:33859
I20260810 22:28:59.843688 6762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:33859 every 8 connection(s)
I20260810 22:28:59.844302 6619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:28:59.852690 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 6619
I20260810 22:28:59.852859 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:28:59.853247 1910 external_mini_cluster.cc:1428] Setting key f7612c54cc0a42d60b93f176a67a42ec
Aug 10 22:28:59 dist-test-slave-njxd krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400939, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:28:59.859172 6595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:59.846130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:59291 (local address 127.1.221.190:42049)
0810 22:28:59.846278 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:28:59.846283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:28:59.847083 (+ 800us) server_negotiation.cc:407] Connection header received
0810 22:28:59.848468 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:28:59.848472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:28:59.848533 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:28:59.848648 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:28:59.849895 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:59.850450 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:59.851638 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:28:59.851883 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:28:59.852002 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:59.854933 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:28:59.854937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:28:59.854940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:28:59.854974 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:28:59.856454 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:59.857175 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:59.857179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:59.857180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:59.857266 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:28:59.857745 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:28:59.857749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:28:59.857750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:28:59.857953 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:28:59.858037 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 22:28:59.858758 (+ 721us) server_negotiation.cc:299] Negotiation successful
0810 22:28:59.858914 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:28:59.860167 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:28:59.846450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:42049 (local address 127.1.221.129:59291)
0810 22:28:59.846922 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 22:28:59.846967 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:28:59.848246 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:28:59.848836 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:28:59.848842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:28:59.849209 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:28:59.849724 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:59.849739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:59.850594 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:59.850603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:59.851504 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:28:59.851513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:28:59.852060 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:28:59.852064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:28:59.852241 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:28:59.852938 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:28:59.852963 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:28:59.854763 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:28:59.856602 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:59.856608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:59.856623 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:59.857046 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:59.857378 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:28:59.857385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:28:59.857388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:28:59.857635 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:28:59.858066 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:28:59.858075 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:28:59.858463 (+ 388us) client_negotiation.cc:769] Sending connection context
0810 22:28:59.858647 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 22:28:59.858963 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":132,"threads_started":1}
I20260810 22:28:59.861507 6763 heartbeater.cc:347] Connected to a master server at 127.1.221.190:42049
I20260810 22:28:59.861778 6763 heartbeater.cc:464] Registering TS with master...
I20260810 22:28:59.862306 6763 heartbeater.cc:511] Master 127.1.221.190:42049 requested a full tablet report, sending...
I20260810 22:28:59.863875 6551 ts_manager.cc:194] Registered new tserver with Master: 1951b1d2cae9451b915906e4c0a0ceb3 (127.1.221.129:33859)
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260810 22:28:59.865078 6551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:59291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:28:59.884898 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42049
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:28:59.997561 6769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:28:59.997900 6769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:28:59.998003 6769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:00.002058 6769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:00.002131 6769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:00.002214 6769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:29:00.007359 6769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:00.008421 6769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:00.009464 6769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:00.018872 6775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:00.019119 6769 server_base.cc:1034] running on GCE node
W20260810 22:29:00.018872 6774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:00.018882 6777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:00.019639 6769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:00.020256 6769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:00.021472 6769 hybrid_clock.cc:648] HybridClock initialized: now 1786400940021454 us; error 26 us; skew 500 ppm
Aug 10 22:29:00 dist-test-slave-njxd krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400940, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:00.024325 6769 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:29:00.025094 6769 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:00.025187 6769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:00.025424 6769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:00.027479 6769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "af288b9d67dc4c9887a067a6c27338ab"
format_stamp: "Formatted at 2026-08-10 22:29:00 on dist-test-slave-njxd"
server_key: "f4169a538c758ab3d3e4f04b0138b006"
server_key_iv: "ae2bdf1a7276d9325cec5f5400d3162d"
server_key_version: "encryptionkey@0"
I20260810 22:29:00.028059 6769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "af288b9d67dc4c9887a067a6c27338ab"
format_stamp: "Formatted at 2026-08-10 22:29:00 on dist-test-slave-njxd"
server_key: "f4169a538c758ab3d3e4f04b0138b006"
server_key_iv: "ae2bdf1a7276d9325cec5f5400d3162d"
server_key_version: "encryptionkey@0"
I20260810 22:29:00.031669 6769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:29:00.034053 6786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:00.035259 6769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:29:00.035404 6769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "af288b9d67dc4c9887a067a6c27338ab"
format_stamp: "Formatted at 2026-08-10 22:29:00 on dist-test-slave-njxd"
server_key: "f4169a538c758ab3d3e4f04b0138b006"
server_key_iv: "ae2bdf1a7276d9325cec5f5400d3162d"
server_key_version: "encryptionkey@0"
I20260810 22:29:00.035512 6769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:00.051679 6769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:00.055161 6769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:00.055375 6769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:00.055989 6769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:00.056833 6769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:00.056906 6769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:00.056980 6769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:00.057021 6769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:00.066215 6769 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:33139
I20260810 22:29:00.066236 6911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:33139 every 8 connection(s)
I20260810 22:29:00.066867 6769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:29:00.072439 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 6769
I20260810 22:29:00.072580 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:29:00.072935 1910 external_mini_cluster.cc:1428] Setting key de3cb079a65fa099f9ceda612b129a2c
Aug 10 22:29:00 dist-test-slave-njxd krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400940, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:29:00.083235 6595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.068705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:52135 (local address 127.1.221.190:42049)
0810 22:29:00.068852 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:00.068856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:00.069839 (+ 983us) server_negotiation.cc:407] Connection header received
0810 22:29:00.071287 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:00.071290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:00.071351 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:00.071457 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:00.072732 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.073458 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.075066 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.075455 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.075587 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.078813 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:00.078820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:00.078826 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:00.078863 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:00.080787 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:00.081363 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:00.081370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:00.081374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:00.081478 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:00.081907 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:00.081913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:00.081918 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:00.082204 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:00.082311 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:00.082860 (+ 549us) server_negotiation.cc:299] Negotiation successful
0810 22:29:00.083014 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:29:00.083973 6914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.069090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:42049 (local address 127.1.221.130:52135)
0810 22:29:00.069683 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:00.069729 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:00.071023 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:00.071642 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:00.071649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:00.072015 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:00.072530 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.072548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.073667 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.073670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.074943 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.074953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.075720 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.075723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.075929 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.076619 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:00.076642 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:00.078599 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:00.080923 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:00.080929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:00.080943 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:00.081235 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:00.081623 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:00.081626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:00.081628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:00.081787 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:00.082319 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:00.082325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:00.082578 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 22:29:00.082737 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 22:29:00.083013 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":182,"threads_started":1}
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260810 22:29:00.085263 6912 heartbeater.cc:347] Connected to a master server at 127.1.221.190:42049
I20260810 22:29:00.085529 6912 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:00.086126 6912 heartbeater.cc:511] Master 127.1.221.190:42049 requested a full tablet report, sending...
I20260810 22:29:00.087215 6551 ts_manager.cc:194] Registered new tserver with Master: af288b9d67dc4c9887a067a6c27338ab (127.1.221.130:33139)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260810 22:29:00.087937 6551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:52135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:29:00.102969 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42049
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:00.216926 6917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:00.217188 6917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:00.217236 6917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:00.221328 6917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:00.221395 6917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:00.221491 6917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:29:00.226989 6917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:00.228070 6917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:00.229120 6917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:00.238708 6924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:00.238719 6928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:00.238698 6925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:00.238910 6917 server_base.cc:1034] running on GCE node
I20260810 22:29:00.239557 6917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:00.240142 6917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:00.241313 6917 hybrid_clock.cc:648] HybridClock initialized: now 1786400940241296 us; error 31 us; skew 500 ppm
Aug 10 22:29:00 dist-test-slave-njxd krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400940, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:00.244252 6917 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:29:00.245002 6917 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:00.245069 6917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:00.245297 6917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:00.247445 6917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "dfc15f6e1df3489880fcba4fd0d09e55"
format_stamp: "Formatted at 2026-08-10 22:29:00 on dist-test-slave-njxd"
server_key: "f83c4fb2ce78a622e57433e4889e83c5"
server_key_iv: "903f4a3029ee4b993e8e9d7aefa4fdc4"
server_key_version: "encryptionkey@0"
I20260810 22:29:00.248049 6917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "dfc15f6e1df3489880fcba4fd0d09e55"
format_stamp: "Formatted at 2026-08-10 22:29:00 on dist-test-slave-njxd"
server_key: "f83c4fb2ce78a622e57433e4889e83c5"
server_key_iv: "903f4a3029ee4b993e8e9d7aefa4fdc4"
server_key_version: "encryptionkey@0"
I20260810 22:29:00.252005 6917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:29:00.254492 6935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:00.255793 6917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260810 22:29:00.255941 6917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "dfc15f6e1df3489880fcba4fd0d09e55"
format_stamp: "Formatted at 2026-08-10 22:29:00 on dist-test-slave-njxd"
server_key: "f83c4fb2ce78a622e57433e4889e83c5"
server_key_iv: "903f4a3029ee4b993e8e9d7aefa4fdc4"
server_key_version: "encryptionkey@0"
I20260810 22:29:00.256058 6917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:00.270720 6917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:00.274447 6917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:00.274686 6917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:00.275385 6917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:00.276367 6917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:00.276446 6917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:00.276521 6917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:00.276566 6917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:00.286278 6917 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:36951
I20260810 22:29:00.286288 7066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:36951 every 8 connection(s)
I20260810 22:29:00.286942 6917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:29:00.289443 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 6917
I20260810 22:29:00.289618 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:29:00.289994 1910 external_mini_cluster.cc:1428] Setting key d2166598e4528c08cf5e19cea2b4a9ef
Aug 10 22:29:00 dist-test-slave-njxd krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400940, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:29:00.300876 6595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.288783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:60205 (local address 127.1.221.190:42049)
0810 22:29:00.288974 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:00.288978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:00.289927 (+ 949us) server_negotiation.cc:407] Connection header received
0810 22:29:00.291594 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:00.291597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:00.291657 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:00.291735 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:00.292992 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.293513 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.294697 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.294910 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.294987 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.297463 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:00.297467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:00.297468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:00.297495 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:00.299006 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:00.299453 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:00.299455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:00.299457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:00.299532 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:00.299830 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:00.299832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:00.299833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:00.300000 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:00.300074 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:00.300583 (+ 509us) server_negotiation.cc:299] Negotiation successful
0810 22:29:00.300708 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 22:29:00.301822 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.289202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:42049 (local address 127.1.221.131:60205)
0810 22:29:00.289738 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:00.289806 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:00.291347 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:00.291905 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:00.291912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:00.292367 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:00.292864 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.292879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.293670 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.293673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.294582 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.294591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.295041 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.295044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.295193 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.295761 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:00.295784 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:00.297315 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:00.299101 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:00.299106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:00.299117 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:00.299366 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:00.299612 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:00.299614 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:00.299615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:00.299743 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:00.300086 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:00.300091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:00.300321 (+ 230us) client_negotiation.cc:769] Sending connection context
0810 22:29:00.300535 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 22:29:00.300878 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":151,"threads_started":1}
I20260810 22:29:00.303153 7067 heartbeater.cc:347] Connected to a master server at 127.1.221.190:42049
I20260810 22:29:00.303414 7067 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:00.304037 7067 heartbeater.cc:511] Master 127.1.221.190:42049 requested a full tablet report, sending...
I20260810 22:29:00.304997 6551 ts_manager.cc:194] Registered new tserver with Master: dfc15f6e1df3489880fcba4fd0d09e55 (127.1.221.131:36951)
I20260810 22:29:00.305656 6551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:60205
I20260810 22:29:00.315706 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:29:00.325439 6595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.317696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.1.221.190:42049)
0810 22:29:00.317867 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:00.317872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:00.318066 (+ 194us) server_negotiation.cc:407] Connection header received
0810 22:29:00.318260 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:00.318263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:00.318315 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:00.318402 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:00.319152 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.319672 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.320524 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.320749 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.320855 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.322357 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:00.322365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:00.322368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:00.322391 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:00.323637 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:00.324064 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:00.324067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:00.324069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:00.324221 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:00.324510 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:00.324514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:00.324516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:00.324750 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:00.324899 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:00.325135 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 22:29:00.325267 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:29:00.328336 6551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:29:00.331471 6551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:29:00.346727 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.340241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36951 (local address 127.0.0.1:56734)
0810 22:29:00.341086 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:00.341142 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:00.341296 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:00.342380 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:00.342385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:00.342408 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:00.342657 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.342664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.344180 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.344183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.345292 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.345303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.346258 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.346261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.346364 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.346397 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 22:29:00.346441 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 22:29:00.346557 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":89,"spinlock_wait_cycles":16384,"thread_start_us":99,"threads_started":1}
I20260810 22:29:00.346753 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.341181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55164 (local address 127.1.221.129:33859)
0810 22:29:00.341568 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:00.341598 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:00.341951 (+ 353us) server_negotiation.cc:407] Connection header received
0810 22:29:00.342033 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:00.342037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:00.342205 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:00.342344 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:00.342813 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.343544 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.345100 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.345962 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.346084 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.346250 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:00.346467 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 22:29:00.346576 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":116,"threads_started":1}
I20260810 22:29:00.347077 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.340755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33859 (local address 127.0.0.1:55164)
0810 22:29:00.341730 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:00.341746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:00.341855 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:00.342512 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:00.342515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:00.342531 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:00.342678 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.342688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.343677 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.343684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.344957 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.344968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.346102 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.346106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.346241 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.346290 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 22:29:00.346405 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 22:29:00.346965 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":30,"thread_start_us":95,"threads_started":1}
I20260810 22:29:00.347385 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.340413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:33139 (local address 127.0.0.1:40596)
0810 22:29:00.341085 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:00.341143 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:00.341294 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:00.342407 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:00.342410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:00.342437 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:00.342636 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.342644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.344148 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.344152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.345245 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.345254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.346528 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.346532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.346631 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.346659 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 22:29:00.346731 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 22:29:00.346825 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":28,"spinlock_wait_cycles":18432,"thread_start_us":119,"threads_started":1}
I20260810 22:29:00.347450 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.341096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40596 (local address 127.1.221.130:33139)
0810 22:29:00.341682 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:00.341694 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:00.341718 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:29:00.341813 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:00.341820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:00.342042 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:00.342193 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:00.342775 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.343990 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.345380 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.346204 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.346316 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.346461 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:00.347170 (+ 709us) server_negotiation.cc:299] Negotiation successful
0810 22:29:00.347292 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":128,"threads_started":1}
I20260810 22:29:00.347710 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.340418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.1.221.131:36951)
0810 22:29:00.341850 (+ 1432us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:00.341864 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:00.341888 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:29:00.341963 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:00.341968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:00.342192 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:00.342385 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:00.342812 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.343990 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.345444 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.346123 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.346528 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.346681 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:00.346732 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 22:29:00.347315 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1320,"thread_start_us":136,"threads_started":1}
I20260810 22:29:00.350342 6990 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb3256bb3164150bfec131c322fe8e6 (DEFAULT_TABLE table=test-table [id=c2b1783070dd460fb6319747a71d6122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:00.350353 6689 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb3256bb3164150bfec131c322fe8e6 (DEFAULT_TABLE table=test-table [id=c2b1783070dd460fb6319747a71d6122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:00.350351 6838 tablet_service.cc:1467] Processing CreateTablet for tablet 2fb3256bb3164150bfec131c322fe8e6 (DEFAULT_TABLE table=test-table [id=c2b1783070dd460fb6319747a71d6122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:00.351402 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb3256bb3164150bfec131c322fe8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:00.351416 6689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb3256bb3164150bfec131c322fe8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:00.351416 6838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fb3256bb3164150bfec131c322fe8e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:00.358911 7087 tablet_bootstrap.cc:492] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55: Bootstrap starting.
I20260810 22:29:00.359562 7088 tablet_bootstrap.cc:492] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab: Bootstrap starting.
I20260810 22:29:00.359562 7089 tablet_bootstrap.cc:492] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3: Bootstrap starting.
I20260810 22:29:00.361428 7087 tablet_bootstrap.cc:654] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:00.361814 7089 tablet_bootstrap.cc:654] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:00.361846 7088 tablet_bootstrap.cc:654] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:00.362718 7087 log.cc:824] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:00.362792 7089 log.cc:824] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:00.363077 7088 log.cc:824] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:00.364964 7089 tablet_bootstrap.cc:492] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3: No bootstrap required, opened a new log
I20260810 22:29:00.365012 7087 tablet_bootstrap.cc:492] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55: No bootstrap required, opened a new log
I20260810 22:29:00.365125 7088 tablet_bootstrap.cc:492] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab: No bootstrap required, opened a new log
I20260810 22:29:00.365209 7087 ts_tablet_manager.cc:1397] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 22:29:00.365217 7089 ts_tablet_manager.cc:1397] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:29:00.365290 7088 ts_tablet_manager.cc:1397] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:29:00.368738 7089 raft_consensus.cc:348] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36951 } } peers { permanent_uuid: "af288b9d67dc4c9887a067a6c27338ab" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33139 } } peers { permanent_uuid: "1951b1d2cae9451b915906e4c0a0ceb3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33859 } }
I20260810 22:29:00.369009 7089 raft_consensus.cc:374] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:00.369064 7089 raft_consensus.cc:729] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1951b1d2cae9451b915906e4c0a0ceb3, State: Initialized, Role: FOLLOWER
I20260810 22:29:00.369181 7088 raft_consensus.cc:348] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36951 } } peers { permanent_uuid: "af288b9d67dc4c9887a067a6c27338ab" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33139 } } peers { permanent_uuid: "1951b1d2cae9451b915906e4c0a0ceb3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33859 } }
I20260810 22:29:00.369181 7087 raft_consensus.cc:348] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36951 } } peers { permanent_uuid: "af288b9d67dc4c9887a067a6c27338ab" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33139 } } peers { permanent_uuid: "1951b1d2cae9451b915906e4c0a0ceb3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33859 } }
I20260810 22:29:00.369468 7088 raft_consensus.cc:374] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:00.369468 7087 raft_consensus.cc:374] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:00.369529 7088 raft_consensus.cc:729] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af288b9d67dc4c9887a067a6c27338ab, State: Initialized, Role: FOLLOWER
I20260810 22:29:00.369536 7087 raft_consensus.cc:729] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc15f6e1df3489880fcba4fd0d09e55, State: Initialized, Role: FOLLOWER
I20260810 22:29:00.369619 7089 consensus_queue.cc:260] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36951 } } peers { permanent_uuid: "af288b9d67dc4c9887a067a6c27338ab" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33139 } } peers { permanent_uuid: "1951b1d2cae9451b915906e4c0a0ceb3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33859 } }
I20260810 22:29:00.370052 7088 consensus_queue.cc:260] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36951 } } peers { permanent_uuid: "af288b9d67dc4c9887a067a6c27338ab" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33139 } } peers { permanent_uuid: "1951b1d2cae9451b915906e4c0a0ceb3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33859 } }
I20260810 22:29:00.370052 7087 consensus_queue.cc:260] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36951 } } peers { permanent_uuid: "af288b9d67dc4c9887a067a6c27338ab" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33139 } } peers { permanent_uuid: "1951b1d2cae9451b915906e4c0a0ceb3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33859 } }
I20260810 22:29:00.370482 6763 heartbeater.cc:503] Master 127.1.221.190:42049 was elected leader, sending a full tablet report...
I20260810 22:29:00.370817 7089 ts_tablet_manager.cc:1428] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:29:00.371038 7088 ts_tablet_manager.cc:1428] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 22:29:00.371187 6912 heartbeater.cc:503] Master 127.1.221.190:42049 was elected leader, sending a full tablet report...
I20260810 22:29:00.372400 7087 ts_tablet_manager.cc:1428] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 22:29:00.372746 7067 heartbeater.cc:503] Master 127.1.221.190:42049 was elected leader, sending a full tablet report...
I20260810 22:29:00.508661 7095 raft_consensus.cc:482] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:29:00.508929 7095 raft_consensus.cc:504] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36951 } } peers { permanent_uuid: "af288b9d67dc4c9887a067a6c27338ab" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33139 } } peers { permanent_uuid: "1951b1d2cae9451b915906e4c0a0ceb3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33859 } }
I20260810 22:29:00.510269 7095 leader_election.cc:288] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af288b9d67dc4c9887a067a6c27338ab (127.1.221.130:33139), 1951b1d2cae9451b915906e4c0a0ceb3 (127.1.221.129:33859)
I20260810 22:29:00.515012 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.510895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:59049 (local address 127.1.221.129:33859)
0810 22:29:00.511064 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:00.511068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:00.511178 (+ 110us) server_negotiation.cc:407] Connection header received
0810 22:29:00.511319 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:00.511321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:00.511387 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:00.511462 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:00.511760 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.512449 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.513866 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.514471 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.514601 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.514644 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:00.514836 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 22:29:00.514902 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 22:29:00.515012 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.510636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:46203 (local address 127.1.221.130:33139)
0810 22:29:00.510814 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:00.510819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:00.510838 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:29:00.510900 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:00.510903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:00.510978 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:00.511071 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:00.511422 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.512204 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.513732 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.514347 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.514472 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.514558 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:00.514803 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 22:29:00.514882 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 22:29:00.515025 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.510594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33859 (local address 127.1.221.131:59049)
0810 22:29:00.511082 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:00.511097 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:00.511204 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:00.511486 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:00.511487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:00.511504 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:00.511632 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.511639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.512583 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.512587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.513761 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.513768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.514614 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.514617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.514720 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.514747 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:29:00.514811 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 22:29:00.514859 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"spinlock_wait_cycles":768,"thread_start_us":133,"threads_started":1}
I20260810 22:29:00.515017 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.510496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:33139 (local address 127.1.221.131:46203)
0810 22:29:00.510650 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:00.510670 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:00.510783 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:00.511113 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:00.511120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:00.511146 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:00.511305 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.511312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.512311 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.512313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.513610 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:00.513621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.514467 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.514469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:00.514608 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.514658 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 22:29:00.514740 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 22:29:00.514828 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":15,"spinlock_wait_cycles":8192}
I20260810 22:29:00.515749 6712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb3256bb3164150bfec131c322fe8e6" candidate_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1951b1d2cae9451b915906e4c0a0ceb3" is_pre_election: true
I20260810 22:29:00.515782 6862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb3256bb3164150bfec131c322fe8e6" candidate_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af288b9d67dc4c9887a067a6c27338ab" is_pre_election: true
I20260810 22:29:00.516067 6712 raft_consensus.cc:2454] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfc15f6e1df3489880fcba4fd0d09e55 in term 0.
I20260810 22:29:00.516086 6862 raft_consensus.cc:2454] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfc15f6e1df3489880fcba4fd0d09e55 in term 0.
I20260810 22:29:00.516626 6937 leader_election.cc:302] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1951b1d2cae9451b915906e4c0a0ceb3, af288b9d67dc4c9887a067a6c27338ab, dfc15f6e1df3489880fcba4fd0d09e55; no voters:
I20260810 22:29:00.516921 7095 raft_consensus.cc:2781] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:29:00.517000 7095 raft_consensus.cc:482] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:29:00.517050 7095 raft_consensus.cc:3037] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:00.518280 7095 raft_consensus.cc:504] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36951 } } peers { permanent_uuid: "af288b9d67dc4c9887a067a6c27338ab" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33139 } } peers { permanent_uuid: "1951b1d2cae9451b915906e4c0a0ceb3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33859 } }
I20260810 22:29:00.518672 7095 leader_election.cc:288] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [CANDIDATE]: Term 1 election: Requested vote from peers af288b9d67dc4c9887a067a6c27338ab (127.1.221.130:33139), 1951b1d2cae9451b915906e4c0a0ceb3 (127.1.221.129:33859)
I20260810 22:29:00.519026 6862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb3256bb3164150bfec131c322fe8e6" candidate_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af288b9d67dc4c9887a067a6c27338ab"
I20260810 22:29:00.519071 6712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fb3256bb3164150bfec131c322fe8e6" candidate_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1951b1d2cae9451b915906e4c0a0ceb3"
I20260810 22:29:00.519186 6862 raft_consensus.cc:3037] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:00.519200 6712 raft_consensus.cc:3037] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:00.520279 6712 raft_consensus.cc:2454] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfc15f6e1df3489880fcba4fd0d09e55 in term 1.
I20260810 22:29:00.520365 6862 raft_consensus.cc:2454] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfc15f6e1df3489880fcba4fd0d09e55 in term 1.
I20260810 22:29:00.520699 6938 leader_election.cc:302] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af288b9d67dc4c9887a067a6c27338ab, dfc15f6e1df3489880fcba4fd0d09e55; no voters:
I20260810 22:29:00.520941 7095 raft_consensus.cc:2781] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:00.521235 7095 raft_consensus.cc:686] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [term 1 LEADER]: Becoming Leader. State: Replica: dfc15f6e1df3489880fcba4fd0d09e55, State: Running, Role: LEADER
I20260810 22:29:00.521667 7095 consensus_queue.cc:237] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36951 } } peers { permanent_uuid: "af288b9d67dc4c9887a067a6c27338ab" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33139 } } peers { permanent_uuid: "1951b1d2cae9451b915906e4c0a0ceb3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33859 } }
I20260810 22:29:00.525110 6548 catalog_manager.cc:5495] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfc15f6e1df3489880fcba4fd0d09e55 (127.1.221.131). New cstate: current_term: 1 leader_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfc15f6e1df3489880fcba4fd0d09e55" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af288b9d67dc4c9887a067a6c27338ab" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1951b1d2cae9451b915906e4c0a0ceb3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33859 } health_report { overall_health: UNKNOWN } } }
W20260810 22:29:00.537858 7068 tablet.cc:2369] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:29:00.538713 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.534900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.1.221.131:36951)
0810 22:29:00.535053 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:00.535057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:00.535073 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:29:00.535174 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:00.535176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:00.535232 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:00.535354 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:29:00.535618 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.536129 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.536981 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:00.537205 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:00.537334 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:00.537707 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:29:00.538286 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:29:00.538415 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:00.538496 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 22:29:00.538600 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 22:29:00.544632 6712 raft_consensus.cc:1261] T 2fb3256bb3164150bfec131c322fe8e6 P 1951b1d2cae9451b915906e4c0a0ceb3 [term 1 FOLLOWER]: Refusing update from remote peer dfc15f6e1df3489880fcba4fd0d09e55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:00.544641 6862 raft_consensus.cc:1261] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab [term 1 FOLLOWER]: Refusing update from remote peer dfc15f6e1df3489880fcba4fd0d09e55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:00.545347 7095 consensus_queue.cc:1035] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af288b9d67dc4c9887a067a6c27338ab" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 33139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:00.545519 7097 consensus_queue.cc:1035] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1951b1d2cae9451b915906e4c0a0ceb3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:00.554141 7100 mvcc.cc:205] Tried to move back new op lower bound from 7317098252463366144 to 7317098252379762688. Current Snapshot: MvccSnapshot[applied={T|T < 7317098252463366144}]
I20260810 22:29:00.555105 7101 mvcc.cc:205] Tried to move back new op lower bound from 7317098252463366144 to 7317098252379762688. Current Snapshot: MvccSnapshot[applied={T|T < 7317098252463366144}]
I20260810 22:29:00.560213 7102 mvcc.cc:205] Tried to move back new op lower bound from 7317098252463366144 to 7317098252379762688. Current Snapshot: MvccSnapshot[applied={T|T < 7317098252463366144}]
I20260810 22:29:00.562199 6548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:00.562382 6548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:00.565105 6548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e32441cb65f4b22bc810f9df9a2b239: Sending DeleteTablet for 3 replicas of tablet 2fb3256bb3164150bfec131c322fe8e6
I20260810 22:29:00.565933 6990 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb3256bb3164150bfec131c322fe8e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:00 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:56734
I20260810 22:29:00.566154 6689 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb3256bb3164150bfec131c322fe8e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:00 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:55164
I20260810 22:29:00.566514 6838 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fb3256bb3164150bfec131c322fe8e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:00 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:40596
I20260810 22:29:00.567081 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 6619
I20260810 22:29:00.574230 7113 tablet_replica.cc:330] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab: stopping tablet replica
I20260810 22:29:00.574690 7113 raft_consensus.cc:2229] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:00.575150 7113 raft_consensus.cc:2258] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:00.575191 7111 tablet_replica.cc:330] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55: stopping tablet replica
I20260810 22:29:00.575532 7111 raft_consensus.cc:2229] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:29:00.576018 7111 raft_consensus.cc:2258] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:00.576951 7113 ts_tablet_manager.cc:1905] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:00.580062 7113 ts_tablet_manager.cc:1918] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:00.580196 7113 log.cc:1196] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/2fb3256bb3164150bfec131c322fe8e6
I20260810 22:29:00.580518 7113 ts_tablet_manager.cc:1939] T 2fb3256bb3164150bfec131c322fe8e6 P af288b9d67dc4c9887a067a6c27338ab: Deleting consensus metadata
I20260810 22:29:00.581845 7111 ts_tablet_manager.cc:1905] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:00.581983 6535 catalog_manager.cc:4841] TS af288b9d67dc4c9887a067a6c27338ab (127.1.221.130:33139): tablet 2fb3256bb3164150bfec131c322fe8e6 (table test-table [id=c2b1783070dd460fb6319747a71d6122]) successfully deleted
W20260810 22:29:00.582710 6534 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:33859 (error 108)
I20260810 22:29:00.583843 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.583500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33859 (local address 127.0.0.1:55166)
0810 22:29:00.583644 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:00.583743 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 22:29:00.584035 6534 catalog_manager.cc:4584] TS 1951b1d2cae9451b915906e4c0a0ceb3 (127.1.221.129:33859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2fb3256bb3164150bfec131c322fe8e6: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33859: connect: Connection refused (error 111)
I20260810 22:29:00.584420 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 6769
I20260810 22:29:00.595116 7111 ts_tablet_manager.cc:1918] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:00.595279 7111 log.cc:1196] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/2fb3256bb3164150bfec131c322fe8e6
I20260810 22:29:00.595665 7111 ts_tablet_manager.cc:1939] T 2fb3256bb3164150bfec131c322fe8e6 P dfc15f6e1df3489880fcba4fd0d09e55: Deleting consensus metadata
I20260810 22:29:00.597267 6536 catalog_manager.cc:4841] TS dfc15f6e1df3489880fcba4fd0d09e55 (127.1.221.131:36951): tablet 2fb3256bb3164150bfec131c322fe8e6 (table test-table [id=c2b1783070dd460fb6319747a71d6122]) successfully deleted
I20260810 22:29:00.605443 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 6917
I20260810 22:29:00.613209 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 6514
2026-08-10T22:29:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2483 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T22:29:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:00Z Disabled control of system clock
I20260810 22:29:00.648581 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:35469
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:35469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 22:29:00.769707 7118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:00.769991 7118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:00.770081 7118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:00.774221 7118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:00.774291 7118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:00.774307 7118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:00.774370 7118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:00.774401 7118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:29:00.779349 7118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:35469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:35469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:00.780606 7118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:00.782138 7118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:00.790848 7126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:00.790879 7118 server_base.cc:1034] running on GCE node
W20260810 22:29:00.790831 7123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:00.790831 7124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:00.791740 7118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:00.792840 7118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:00.794047 7118 hybrid_clock.cc:648] HybridClock initialized: now 1786400940794043 us; error 36 us; skew 500 ppm
I20260810 22:29:00.796008 7118 webserver.cc:460] Webserver started at http://127.1.221.190:36991/ using document root <none> and password file <none>
I20260810 22:29:00.796605 7118 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:00.796669 7118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:00.796926 7118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:00.799053 7118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "efbc12b7bb654e6087ab42c6d51f7662"
format_stamp: "Formatted at 2026-08-10 22:29:00 on dist-test-slave-njxd"
server_key: "ecdb6eca390acd4f7bca9ae9a5a86339"
server_key_iv: "41a3857cedabbb8829b5a45155b3c20e"
server_key_version: "encryptionkey@0"
I20260810 22:29:00.799664 7118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "efbc12b7bb654e6087ab42c6d51f7662"
format_stamp: "Formatted at 2026-08-10 22:29:00 on dist-test-slave-njxd"
server_key: "ecdb6eca390acd4f7bca9ae9a5a86339"
server_key_iv: "41a3857cedabbb8829b5a45155b3c20e"
server_key_version: "encryptionkey@0"
I20260810 22:29:00.803529 7118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:29:00.806056 7134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:00.807337 7118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:29:00.807483 7118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "efbc12b7bb654e6087ab42c6d51f7662"
format_stamp: "Formatted at 2026-08-10 22:29:00 on dist-test-slave-njxd"
server_key: "ecdb6eca390acd4f7bca9ae9a5a86339"
server_key_iv: "41a3857cedabbb8829b5a45155b3c20e"
server_key_version: "encryptionkey@0"
I20260810 22:29:00.807605 7118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:00.812419 7118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:00.813078 7118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:00.813277 7118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:00.820488 7118 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:35469
I20260810 22:29:00.820502 7191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:35469 every 8 connection(s)
I20260810 22:29:00.821460 7118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:29:00.824930 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 7118
I20260810 22:29:00.824955 7192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:00.825079 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:29:00.825385 1910 external_mini_cluster.cc:1428] Setting key c6f144e01320e76551e0b0c38f824913
I20260810 22:29:00.831341 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:00.826876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.1.221.190:35469)
0810 22:29:00.827540 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:00.827557 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:00.827595 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:29:00.828413 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:00.828435 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:00.828813 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:00.829106 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:00.830026 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:00.830031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:00.830043 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:00.830054 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:00.830111 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:00.830127 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:00.830258 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:00.830363 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 22:29:00.830616 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":151,"threads_started":1}
I20260810 22:29:00.832304 7192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662: Bootstrap starting.
I20260810 22:29:00.834658 7192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:00.835431 7192 log.cc:824] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:00.837378 7192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662: No bootstrap required, opened a new log
I20260810 22:29:00.840895 7192 raft_consensus.cc:348] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efbc12b7bb654e6087ab42c6d51f7662" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35469 } }
I20260810 22:29:00.841150 7192 raft_consensus.cc:374] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:00.841233 7192 raft_consensus.cc:729] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbc12b7bb654e6087ab42c6d51f7662, State: Initialized, Role: FOLLOWER
I20260810 22:29:00.841719 7192 consensus_queue.cc:260] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efbc12b7bb654e6087ab42c6d51f7662" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35469 } }
I20260810 22:29:00.841884 7192 raft_consensus.cc:388] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:29:00.841974 7192 raft_consensus.cc:482] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:29:00.842073 7192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:00.843010 7192 raft_consensus.cc:504] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efbc12b7bb654e6087ab42c6d51f7662" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35469 } }
I20260810 22:29:00.843353 7192 leader_election.cc:302] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efbc12b7bb654e6087ab42c6d51f7662; no voters:
I20260810 22:29:00.843708 7192 leader_election.cc:288] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:29:00.843873 7197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:00.844125 7197 raft_consensus.cc:686] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [term 1 LEADER]: Becoming Leader. State: Replica: efbc12b7bb654e6087ab42c6d51f7662, State: Running, Role: LEADER
I20260810 22:29:00.844452 7197 consensus_queue.cc:237] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efbc12b7bb654e6087ab42c6d51f7662" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35469 } }
I20260810 22:29:00.845045 7192 sys_catalog.cc:564] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:00.846194 7199 sys_catalog.cc:455] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efbc12b7bb654e6087ab42c6d51f7662. Latest consensus state: current_term: 1 leader_uuid: "efbc12b7bb654e6087ab42c6d51f7662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efbc12b7bb654e6087ab42c6d51f7662" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35469 } } }
I20260810 22:29:00.846222 7198 sys_catalog.cc:455] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efbc12b7bb654e6087ab42c6d51f7662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efbc12b7bb654e6087ab42c6d51f7662" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35469 } } }
I20260810 22:29:00.846349 7199 sys_catalog.cc:458] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:00.846349 7198 sys_catalog.cc:458] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662 [sys.catalog]: This master's current role is: LEADER
W20260810 22:29:00.850235 7212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:29:00.850338 7212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:29:00.850420 7206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:00.851121 7206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:00.856051 7206 catalog_manager.cc:1347] Generated new cluster ID: 2cff275cab6f4fc5b7d5d0327cfbe6a9
I20260810 22:29:00.856118 7206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:00.870102 7206 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:29:00.870465 7206 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:00.888280 7206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662: Generated new TSK 0
I20260810 22:29:00.889083 7206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:29:00.900780 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:35469
--builtin_ntp_servers=127.1.221.148:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 22:29:01.017297 7216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:01.017673 7216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:01.017733 7216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:01.021806 7216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:01.021864 7216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:01.021921 7216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:29:01.026844 7216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:35469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:01.028117 7216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:01.029538 7216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:01.039618 7222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:01.039644 7225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:01.039686 7221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:01.040115 7216 server_base.cc:1034] running on GCE node
I20260810 22:29:01.040546 7216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:01.041247 7216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:01.042517 7216 hybrid_clock.cc:648] HybridClock initialized: now 1786400941042508 us; error 53 us; skew 500 ppm
I20260810 22:29:01.044595 7216 webserver.cc:460] Webserver started at http://127.1.221.129:43231/ using document root <none> and password file <none>
I20260810 22:29:01.045253 7216 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:01.045362 7216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:01.045691 7216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:01.048103 7216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "d03660fdd4aa4b61b2012134cdbf9425"
format_stamp: "Formatted at 2026-08-10 22:29:01 on dist-test-slave-njxd"
server_key: "28e32935945c2c0f4cbdafc046648c50"
server_key_iv: "b49a7030428e65d50591704193f6cd56"
server_key_version: "encryptionkey@0"
I20260810 22:29:01.048720 7216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "d03660fdd4aa4b61b2012134cdbf9425"
format_stamp: "Formatted at 2026-08-10 22:29:01 on dist-test-slave-njxd"
server_key: "28e32935945c2c0f4cbdafc046648c50"
server_key_iv: "b49a7030428e65d50591704193f6cd56"
server_key_version: "encryptionkey@0"
I20260810 22:29:01.052644 7216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:29:01.055301 7231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:01.056668 7216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:29:01.056830 7216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "d03660fdd4aa4b61b2012134cdbf9425"
format_stamp: "Formatted at 2026-08-10 22:29:01 on dist-test-slave-njxd"
server_key: "28e32935945c2c0f4cbdafc046648c50"
server_key_iv: "b49a7030428e65d50591704193f6cd56"
server_key_version: "encryptionkey@0"
I20260810 22:29:01.056967 7216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:01.062078 7216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:01.062774 7216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:01.062983 7216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:01.063592 7216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:01.064572 7216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:01.064661 7216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:01.064723 7216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:01.064775 7216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:01.076648 7216 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:41467
I20260810 22:29:01.076691 7355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:41467 every 8 connection(s)
I20260810 22:29:01.077934 7216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:29:01.078366 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 7216
I20260810 22:29:01.078487 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:29:01.078816 1910 external_mini_cluster.cc:1428] Setting key 02c9031fbe760625669785ea6c4ea67a
I20260810 22:29:01.081208 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:35469
--builtin_ntp_servers=127.1.221.148:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 22:29:01.085289 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.080014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:56629 (local address 127.1.221.190:35469)
0810 22:29:01.080156 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:01.080160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:01.081567 (+ 1407us) server_negotiation.cc:407] Connection header received
0810 22:29:01.083213 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:01.083216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:01.083258 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:01.083332 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:01.084720 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:01.084726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:01.084732 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:01.084752 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:01.084782 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:01.084797 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:01.084913 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:01.085016 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 22:29:01.085072 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:29:01.086223 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.080601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35469 (local address 127.1.221.129:56629)
0810 22:29:01.081381 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:01.081440 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:01.082940 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:01.083548 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:01.083561 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:01.083699 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:01.084382 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:01.084384 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:01.084450 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:01.084540 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:01.084883 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:01.084888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:01.084892 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 22:29:01.085030 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 22:29:01.085250 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":160,"threads_started":1}
I20260810 22:29:01.087675 7356 heartbeater.cc:347] Connected to a master server at 127.1.221.190:35469
I20260810 22:29:01.087942 7356 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:01.088390 7356 heartbeater.cc:511] Master 127.1.221.190:35469 requested a full tablet report, sending...
I20260810 22:29:01.090052 7151 ts_manager.cc:194] Registered new tserver with Master: d03660fdd4aa4b61b2012134cdbf9425 (127.1.221.129:41467)
W20260810 22:29:01.197193 7359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:01.197476 7359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:01.197518 7359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:01.201795 7359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:01.201851 7359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:01.201910 7359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:29:01.206701 7359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:35469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:01.207803 7359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:01.209290 7359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:01.219153 7365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:01.219173 7359 server_base.cc:1034] running on GCE node
W20260810 22:29:01.219362 7367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:01.219220 7364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:01.219877 7359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:01.220546 7359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:01.221755 7359 hybrid_clock.cc:648] HybridClock initialized: now 1786400941221736 us; error 29 us; skew 500 ppm
I20260810 22:29:01.223644 7359 webserver.cc:460] Webserver started at http://127.1.221.130:44959/ using document root <none> and password file <none>
I20260810 22:29:01.224275 7359 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:01.224377 7359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:01.224603 7359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:01.226907 7359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "c24de3341760467d97c2864931b16736"
format_stamp: "Formatted at 2026-08-10 22:29:01 on dist-test-slave-njxd"
server_key: "21ba8be91400b8f699ee9aec55da1422"
server_key_iv: "1b5b30543f5112f37f3a2f420583c4d9"
server_key_version: "encryptionkey@0"
I20260810 22:29:01.227519 7359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "c24de3341760467d97c2864931b16736"
format_stamp: "Formatted at 2026-08-10 22:29:01 on dist-test-slave-njxd"
server_key: "21ba8be91400b8f699ee9aec55da1422"
server_key_iv: "1b5b30543f5112f37f3a2f420583c4d9"
server_key_version: "encryptionkey@0"
I20260810 22:29:01.231583 7359 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:29:01.234248 7374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:01.235592 7359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:29:01.235759 7359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "c24de3341760467d97c2864931b16736"
format_stamp: "Formatted at 2026-08-10 22:29:01 on dist-test-slave-njxd"
server_key: "21ba8be91400b8f699ee9aec55da1422"
server_key_iv: "1b5b30543f5112f37f3a2f420583c4d9"
server_key_version: "encryptionkey@0"
I20260810 22:29:01.235882 7359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:01.240690 7359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:01.241303 7359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:01.241492 7359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:01.242105 7359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:01.243049 7359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:01.243139 7359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:01.243229 7359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:01.243279 7359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:01.252449 7359 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:36489
I20260810 22:29:01.252472 7500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:36489 every 8 connection(s)
I20260810 22:29:01.253412 7359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:29:01.259573 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 7359
I20260810 22:29:01.259737 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:29:01.260058 1910 external_mini_cluster.cc:1428] Setting key 0b90a1c33e2a92dcb3c4b0c67ff03e08
I20260810 22:29:01.260061 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.255415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:37919 (local address 127.1.221.190:35469)
0810 22:29:01.255600 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:01.255605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:01.256775 (+ 1170us) server_negotiation.cc:407] Connection header received
0810 22:29:01.258277 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:01.258280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:01.258323 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:01.258404 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:01.259527 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:01.259530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:01.259533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:01.259547 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:01.259574 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:01.259589 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:01.259698 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:01.259862 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 22:29:01.259914 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 22:29:01.261186 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.255884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35469 (local address 127.1.221.130:37919)
0810 22:29:01.256596 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:01.256642 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:01.258045 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:01.258583 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:01.258589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:01.258681 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:01.259247 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:01.259248 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:01.259291 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:01.259364 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:01.259679 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:01.259689 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:01.259695 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:29:01.259899 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 22:29:01.260155 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":132,"threads_started":1}
I20260810 22:29:01.262398 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:35469
--builtin_ntp_servers=127.1.221.148:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 22:29:01.262471 7501 heartbeater.cc:347] Connected to a master server at 127.1.221.190:35469
I20260810 22:29:01.262713 7501 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:01.263219 7501 heartbeater.cc:511] Master 127.1.221.190:35469 requested a full tablet report, sending...
I20260810 22:29:01.264431 7151 ts_manager.cc:194] Registered new tserver with Master: c24de3341760467d97c2864931b16736 (127.1.221.130:36489)
W20260810 22:29:01.380246 7504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:01.380541 7504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:01.380584 7504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:01.384678 7504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:01.384747 7504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:01.384805 7504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:29:01.389596 7504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:35469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:01.390781 7504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:01.392120 7504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:01.401945 7512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:01.402004 7510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:01.401945 7509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:01.402220 7504 server_base.cc:1034] running on GCE node
I20260810 22:29:01.402756 7504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:01.403467 7504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:01.404668 7504 hybrid_clock.cc:648] HybridClock initialized: now 1786400941404654 us; error 32 us; skew 500 ppm
I20260810 22:29:01.406594 7504 webserver.cc:460] Webserver started at http://127.1.221.131:36071/ using document root <none> and password file <none>
I20260810 22:29:01.407203 7504 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:01.407311 7504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:01.407553 7504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:01.409826 7504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "e5efc4cbdbac49db80aa6c39a05f921a"
format_stamp: "Formatted at 2026-08-10 22:29:01 on dist-test-slave-njxd"
server_key: "9a322fb4ccfcb29dca1345f30a417fae"
server_key_iv: "f980d64f6021e1b5f0fa4c599bab7956"
server_key_version: "encryptionkey@0"
I20260810 22:29:01.410435 7504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "e5efc4cbdbac49db80aa6c39a05f921a"
format_stamp: "Formatted at 2026-08-10 22:29:01 on dist-test-slave-njxd"
server_key: "9a322fb4ccfcb29dca1345f30a417fae"
server_key_iv: "f980d64f6021e1b5f0fa4c599bab7956"
server_key_version: "encryptionkey@0"
I20260810 22:29:01.414584 7504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:29:01.417385 7519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:01.418787 7504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:29:01.418941 7504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "e5efc4cbdbac49db80aa6c39a05f921a"
format_stamp: "Formatted at 2026-08-10 22:29:01 on dist-test-slave-njxd"
server_key: "9a322fb4ccfcb29dca1345f30a417fae"
server_key_iv: "f980d64f6021e1b5f0fa4c599bab7956"
server_key_version: "encryptionkey@0"
I20260810 22:29:01.419072 7504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:01.423954 7504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:01.424618 7504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:01.424819 7504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:01.425405 7504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:01.426482 7504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:01.426558 7504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:01.426622 7504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:01.426687 7504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:01.436807 7504 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:45213
I20260810 22:29:01.436826 7643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:45213 every 8 connection(s)
I20260810 22:29:01.437855 7504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:29:01.440466 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 7504
I20260810 22:29:01.440608 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:29:01.440913 1910 external_mini_cluster.cc:1428] Setting key b018059ee6d698b7e0396fd9206b5584
I20260810 22:29:01.444721 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.439858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:37077 (local address 127.1.221.190:35469)
0810 22:29:01.440049 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:01.440056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:01.441241 (+ 1185us) server_negotiation.cc:407] Connection header received
0810 22:29:01.442954 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:01.442959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:01.443006 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:01.443093 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:01.444256 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:01.444258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:01.444261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:01.444272 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:01.444291 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:01.444300 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:01.444372 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:01.444505 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 22:29:01.444562 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:29:01.445786 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.440328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35469 (local address 127.1.221.131:37077)
0810 22:29:01.441080 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:01.441135 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:01.442704 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:01.443302 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:01.443309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:01.443420 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:01.443998 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:01.443999 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:01.444040 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:01.444110 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:01.444377 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:01.444384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:01.444389 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 22:29:01.444523 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 22:29:01.444839 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":155,"threads_started":1}
I20260810 22:29:01.446882 7644 heartbeater.cc:347] Connected to a master server at 127.1.221.190:35469
I20260810 22:29:01.447122 7644 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:01.447593 7644 heartbeater.cc:511] Master 127.1.221.190:35469 requested a full tablet report, sending...
I20260810 22:29:01.448586 7151 ts_manager.cc:194] Registered new tserver with Master: e5efc4cbdbac49db80aa6c39a05f921a (127.1.221.131:45213)
I20260810 22:29:01.454847 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:29:01.458463 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.456768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.1.221.190:35469)
0810 22:29:01.456904 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:01.456908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:01.457081 (+ 173us) server_negotiation.cc:407] Connection header received
0810 22:29:01.457166 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:01.457168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:01.457200 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:01.457265 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:01.458053 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:01.458056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:01.458058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:01.458071 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:01.458095 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:01.458103 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:01.458193 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:01.458267 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 22:29:01.458322 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:29:01.461162 7151 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:29:01.464318 7151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:29:01.476366 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.472849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:45213 (local address 127.0.0.1:55330)
0810 22:29:01.473231 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:01.473291 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:01.473389 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:01.474769 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:01.474772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:01.474837 (+ 65us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:01.475510 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:01.475512 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:01.475532 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:01.475609 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:01.475937 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:01.475945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:01.475949 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 22:29:01.476081 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 22:29:01.476155 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":105,"threads_started":1}
I20260810 22:29:01.477074 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.473692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.1.221.131:45213)
0810 22:29:01.474223 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:01.474235 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:01.474347 (+ 112us) server_negotiation.cc:407] Connection header received
0810 22:29:01.474409 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:01.474417 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:01.474598 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:01.474761 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:01.475730 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:01.475737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:01.475749 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:01.475762 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:01.475810 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:01.475845 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:01.475962 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:01.476118 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 22:29:01.476196 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":156,"threads_started":1}
I20260810 22:29:01.477452 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.473484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36489 (local address 127.0.0.1:57396)
0810 22:29:01.475533 (+ 2049us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:01.475547 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:01.475629 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:01.476048 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:01.476053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:01.476070 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:01.476718 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:01.476720 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:01.476730 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:01.476756 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:01.477187 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:01.477190 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:01.477193 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 22:29:01.477297 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 22:29:01.477333 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1984,"thread_start_us":128,"threads_started":1}
I20260810 22:29:01.477591 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.473950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.1.221.130:36489)
0810 22:29:01.474409 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:01.474420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:01.475654 (+ 1234us) server_negotiation.cc:407] Connection header received
0810 22:29:01.475729 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:01.475734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:01.475913 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:01.476079 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:01.476915 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:01.476924 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:01.476938 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:01.476952 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:01.477001 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:01.477040 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:01.477144 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:01.477321 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 22:29:01.477393 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":131,"threads_started":1}
I20260810 22:29:01.478466 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.472985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.1.221.129:41467)
0810 22:29:01.473693 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:01.473706 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:01.474355 (+ 649us) server_negotiation.cc:407] Connection header received
0810 22:29:01.474410 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:01.474414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:01.474587 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:01.474727 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:01.476929 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:01.476935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:01.476948 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:01.476963 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:01.477010 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:01.477052 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:01.477160 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:01.477368 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 22:29:01.477445 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":173,"threads_started":1}
I20260810 22:29:01.478858 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.472854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41467 (local address 127.0.0.1:42298)
0810 22:29:01.474150 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:01.474165 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:01.474246 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:01.474750 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:01.474756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:01.474837 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:01.476735 (+ 1898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:01.476737 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:01.476748 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:01.476776 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:01.477246 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:01.477249 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:01.477252 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 22:29:01.478692 (+ 1440us) client_negotiation.cc:240] Negotiation successful
0810 22:29:01.478731 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
I20260810 22:29:01.480378 7570 tablet_service.cc:1467] Processing CreateTablet for tablet ae4bd63c4007403097cd0d9689c33676 (DEFAULT_TABLE table=test-table [id=1a10d59ca57647dc8c531fc58357c429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:01.480379 7427 tablet_service.cc:1467] Processing CreateTablet for tablet ae4bd63c4007403097cd0d9689c33676 (DEFAULT_TABLE table=test-table [id=1a10d59ca57647dc8c531fc58357c429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:01.481431 7570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4bd63c4007403097cd0d9689c33676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:01.481469 7427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4bd63c4007403097cd0d9689c33676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:01.481469 7278 tablet_service.cc:1467] Processing CreateTablet for tablet ae4bd63c4007403097cd0d9689c33676 (DEFAULT_TABLE table=test-table [id=1a10d59ca57647dc8c531fc58357c429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:01.482568 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae4bd63c4007403097cd0d9689c33676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:01.487874 7662 tablet_bootstrap.cc:492] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a: Bootstrap starting.
I20260810 22:29:01.489639 7664 tablet_bootstrap.cc:492] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736: Bootstrap starting.
I20260810 22:29:01.489693 7665 tablet_bootstrap.cc:492] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425: Bootstrap starting.
I20260810 22:29:01.491195 7662 tablet_bootstrap.cc:654] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:01.491526 7664 tablet_bootstrap.cc:654] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:01.491782 7665 tablet_bootstrap.cc:654] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:01.492117 7662 log.cc:824] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:01.492301 7664 log.cc:824] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:01.492580 7665 log.cc:824] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:01.494314 7662 tablet_bootstrap.cc:492] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a: No bootstrap required, opened a new log
I20260810 22:29:01.494318 7664 tablet_bootstrap.cc:492] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736: No bootstrap required, opened a new log
I20260810 22:29:01.494427 7665 tablet_bootstrap.cc:492] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425: No bootstrap required, opened a new log
I20260810 22:29:01.494509 7662 ts_tablet_manager.cc:1397] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 22:29:01.494509 7664 ts_tablet_manager.cc:1397] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260810 22:29:01.494585 7665 ts_tablet_manager.cc:1397] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:29:01.497993 7664 raft_consensus.cc:348] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03660fdd4aa4b61b2012134cdbf9425" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41467 } } peers { permanent_uuid: "e5efc4cbdbac49db80aa6c39a05f921a" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45213 } } peers { permanent_uuid: "c24de3341760467d97c2864931b16736" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36489 } }
I20260810 22:29:01.498250 7664 raft_consensus.cc:374] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:01.498315 7664 raft_consensus.cc:729] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24de3341760467d97c2864931b16736, State: Initialized, Role: FOLLOWER
I20260810 22:29:01.498769 7664 consensus_queue.cc:260] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03660fdd4aa4b61b2012134cdbf9425" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41467 } } peers { permanent_uuid: "e5efc4cbdbac49db80aa6c39a05f921a" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45213 } } peers { permanent_uuid: "c24de3341760467d97c2864931b16736" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36489 } }
I20260810 22:29:01.499223 7662 raft_consensus.cc:348] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03660fdd4aa4b61b2012134cdbf9425" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41467 } } peers { permanent_uuid: "e5efc4cbdbac49db80aa6c39a05f921a" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45213 } } peers { permanent_uuid: "c24de3341760467d97c2864931b16736" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36489 } }
I20260810 22:29:01.499194 7665 raft_consensus.cc:348] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03660fdd4aa4b61b2012134cdbf9425" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41467 } } peers { permanent_uuid: "e5efc4cbdbac49db80aa6c39a05f921a" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45213 } } peers { permanent_uuid: "c24de3341760467d97c2864931b16736" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36489 } }
I20260810 22:29:01.499454 7662 raft_consensus.cc:374] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:01.499454 7665 raft_consensus.cc:374] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:01.499461 7501 heartbeater.cc:503] Master 127.1.221.190:35469 was elected leader, sending a full tablet report...
I20260810 22:29:01.499503 7662 raft_consensus.cc:729] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5efc4cbdbac49db80aa6c39a05f921a, State: Initialized, Role: FOLLOWER
I20260810 22:29:01.499503 7665 raft_consensus.cc:729] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03660fdd4aa4b61b2012134cdbf9425, State: Initialized, Role: FOLLOWER
I20260810 22:29:01.500038 7664 ts_tablet_manager.cc:1428] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260810 22:29:01.499964 7665 consensus_queue.cc:260] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03660fdd4aa4b61b2012134cdbf9425" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41467 } } peers { permanent_uuid: "e5efc4cbdbac49db80aa6c39a05f921a" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45213 } } peers { permanent_uuid: "c24de3341760467d97c2864931b16736" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36489 } }
I20260810 22:29:01.499964 7662 consensus_queue.cc:260] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03660fdd4aa4b61b2012134cdbf9425" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41467 } } peers { permanent_uuid: "e5efc4cbdbac49db80aa6c39a05f921a" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45213 } } peers { permanent_uuid: "c24de3341760467d97c2864931b16736" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36489 } }
I20260810 22:29:01.500645 7356 heartbeater.cc:503] Master 127.1.221.190:35469 was elected leader, sending a full tablet report...
I20260810 22:29:01.500957 7662 ts_tablet_manager.cc:1428] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:29:01.500957 7665 ts_tablet_manager.cc:1428] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 22:29:01.501276 7644 heartbeater.cc:503] Master 127.1.221.190:35469 was elected leader, sending a full tablet report...
W20260810 22:29:01.504768 7502 tablet.cc:2369] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:29:01.579511 7357 tablet.cc:2369] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:29:01.609736 7671 raft_consensus.cc:482] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:29:01.610013 7671 raft_consensus.cc:504] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03660fdd4aa4b61b2012134cdbf9425" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41467 } } peers { permanent_uuid: "e5efc4cbdbac49db80aa6c39a05f921a" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45213 } } peers { permanent_uuid: "c24de3341760467d97c2864931b16736" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36489 } }
I20260810 22:29:01.611351 7671 leader_election.cc:288] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5efc4cbdbac49db80aa6c39a05f921a (127.1.221.131:45213), c24de3341760467d97c2864931b16736 (127.1.221.130:36489)
I20260810 22:29:01.613811 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.611767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:56103 (local address 127.1.221.131:45213)
0810 22:29:01.611913 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:01.611918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:01.612130 (+ 212us) server_negotiation.cc:407] Connection header received
0810 22:29:01.612339 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:01.612343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:01.612389 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:01.612472 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:01.613302 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:01.613306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:01.613309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:01.613322 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:01.613345 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:01.613357 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:01.613458 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:01.613550 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 22:29:01.613645 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:29:01.613830 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.611634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:45213 (local address 127.1.221.129:56103)
0810 22:29:01.611992 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:01.612006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:01.612215 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:01.612486 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:01.612489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:01.612503 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:01.613094 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:01.613099 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:01.613114 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:01.613151 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:01.613449 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:01.613453 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:01.613456 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 22:29:01.613564 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 22:29:01.613635 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":100,"threads_started":1}
I20260810 22:29:01.613891 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.611746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:43555 (local address 127.1.221.130:36489)
0810 22:29:01.611909 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:01.611915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:01.612350 (+ 435us) server_negotiation.cc:407] Connection header received
0810 22:29:01.612442 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:01.612447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:01.612495 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:01.612590 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:01.613470 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:01.613473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:01.613476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:01.613492 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:01.613519 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:01.613532 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:01.613652 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:01.613727 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 22:29:01.613763 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:29:01.614166 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.611634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36489 (local address 127.1.221.129:43555)
0810 22:29:01.612220 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:01.612232 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:01.612348 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:01.612609 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:01.612612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:01.612649 (+ 37us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:01.613290 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:01.613292 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:01.613303 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:01.613330 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:01.613639 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:01.613642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:01.613646 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 22:29:01.613732 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 22:29:01.613766 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":298,"thread_start_us":104,"threads_started":1}
I20260810 22:29:01.614740 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4bd63c4007403097cd0d9689c33676" candidate_uuid: "d03660fdd4aa4b61b2012134cdbf9425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5efc4cbdbac49db80aa6c39a05f921a" is_pre_election: true
I20260810 22:29:01.614977 7450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4bd63c4007403097cd0d9689c33676" candidate_uuid: "d03660fdd4aa4b61b2012134cdbf9425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24de3341760467d97c2864931b16736" is_pre_election: true
I20260810 22:29:01.615129 7592 raft_consensus.cc:2454] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d03660fdd4aa4b61b2012134cdbf9425 in term 0.
I20260810 22:29:01.615309 7450 raft_consensus.cc:2454] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d03660fdd4aa4b61b2012134cdbf9425 in term 0.
I20260810 22:29:01.615738 7235 leader_election.cc:302] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c24de3341760467d97c2864931b16736, d03660fdd4aa4b61b2012134cdbf9425, e5efc4cbdbac49db80aa6c39a05f921a; no voters:
I20260810 22:29:01.616072 7671 raft_consensus.cc:2781] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:29:01.616178 7671 raft_consensus.cc:482] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:29:01.616240 7671 raft_consensus.cc:3037] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:01.617383 7671 raft_consensus.cc:504] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03660fdd4aa4b61b2012134cdbf9425" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41467 } } peers { permanent_uuid: "e5efc4cbdbac49db80aa6c39a05f921a" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45213 } } peers { permanent_uuid: "c24de3341760467d97c2864931b16736" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36489 } }
I20260810 22:29:01.617846 7671 leader_election.cc:288] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [CANDIDATE]: Term 1 election: Requested vote from peers e5efc4cbdbac49db80aa6c39a05f921a (127.1.221.131:45213), c24de3341760467d97c2864931b16736 (127.1.221.130:36489)
I20260810 22:29:01.618188 7592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4bd63c4007403097cd0d9689c33676" candidate_uuid: "d03660fdd4aa4b61b2012134cdbf9425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5efc4cbdbac49db80aa6c39a05f921a"
I20260810 22:29:01.618227 7450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae4bd63c4007403097cd0d9689c33676" candidate_uuid: "d03660fdd4aa4b61b2012134cdbf9425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24de3341760467d97c2864931b16736"
I20260810 22:29:01.618307 7592 raft_consensus.cc:3037] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:01.618376 7450 raft_consensus.cc:3037] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:01.619468 7592 raft_consensus.cc:2454] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03660fdd4aa4b61b2012134cdbf9425 in term 1.
I20260810 22:29:01.619503 7450 raft_consensus.cc:2454] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03660fdd4aa4b61b2012134cdbf9425 in term 1.
I20260810 22:29:01.619800 7235 leader_election.cc:302] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d03660fdd4aa4b61b2012134cdbf9425, e5efc4cbdbac49db80aa6c39a05f921a; no voters:
I20260810 22:29:01.620010 7671 raft_consensus.cc:2781] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:01.620265 7671 raft_consensus.cc:686] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [term 1 LEADER]: Becoming Leader. State: Replica: d03660fdd4aa4b61b2012134cdbf9425, State: Running, Role: LEADER
I20260810 22:29:01.620615 7671 consensus_queue.cc:237] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03660fdd4aa4b61b2012134cdbf9425" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41467 } } peers { permanent_uuid: "e5efc4cbdbac49db80aa6c39a05f921a" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45213 } } peers { permanent_uuid: "c24de3341760467d97c2864931b16736" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36489 } }
I20260810 22:29:01.624075 7151 catalog_manager.cc:5495] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 reported cstate change: term changed from 0 to 1, leader changed from <none> to d03660fdd4aa4b61b2012134cdbf9425 (127.1.221.129). New cstate: current_term: 1 leader_uuid: "d03660fdd4aa4b61b2012134cdbf9425" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03660fdd4aa4b61b2012134cdbf9425" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5efc4cbdbac49db80aa6c39a05f921a" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24de3341760467d97c2864931b16736" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36489 } health_report { overall_health: UNKNOWN } } }
I20260810 22:29:01.667703 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.666074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.1.221.129:41467)
0810 22:29:01.666223 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:01.666227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:01.666242 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:29:01.666423 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:01.666426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:01.666495 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:01.666569 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:01.667260 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:01.667263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:01.667265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:01.667277 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:01.667296 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:01.667305 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:01.667384 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:01.667440 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 22:29:01.667475 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:29:01.674690 7450 raft_consensus.cc:1261] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736 [term 1 FOLLOWER]: Refusing update from remote peer d03660fdd4aa4b61b2012134cdbf9425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:01.674734 7592 raft_consensus.cc:1261] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a [term 1 FOLLOWER]: Refusing update from remote peer d03660fdd4aa4b61b2012134cdbf9425: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:01.675524 7675 consensus_queue.cc:1035] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24de3341760467d97c2864931b16736" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:01.675840 7671 consensus_queue.cc:1035] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5efc4cbdbac49db80aa6c39a05f921a" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 45213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:01.684837 7680 mvcc.cc:205] Tried to move back new op lower bound from 7317098257090134016 to 7317098256881004544. Current Snapshot: MvccSnapshot[applied={T|T < 7317098257090134016}]
I20260810 22:29:01.685334 7679 mvcc.cc:205] Tried to move back new op lower bound from 7317098257090134016 to 7317098256881004544. Current Snapshot: MvccSnapshot[applied={T|T < 7317098257090134016}]
I20260810 22:29:01.692456 7151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:01.692657 7151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:01.695132 7151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efbc12b7bb654e6087ab42c6d51f7662: Sending DeleteTablet for 3 replicas of tablet ae4bd63c4007403097cd0d9689c33676
I20260810 22:29:01.695969 7570 tablet_service.cc:1514] Processing DeleteTablet for tablet ae4bd63c4007403097cd0d9689c33676 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:01 UTC) from {username='slave'} at 127.0.0.1:55330
I20260810 22:29:01.695969 7427 tablet_service.cc:1514] Processing DeleteTablet for tablet ae4bd63c4007403097cd0d9689c33676 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:01 UTC) from {username='slave'} at 127.0.0.1:57396
I20260810 22:29:01.695990 7278 tablet_service.cc:1514] Processing DeleteTablet for tablet ae4bd63c4007403097cd0d9689c33676 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:01 UTC) from {username='slave'} at 127.0.0.1:42298
I20260810 22:29:01.696565 7689 tablet_replica.cc:330] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736: stopping tablet replica
I20260810 22:29:01.696646 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 7216
I20260810 22:29:01.696856 7688 tablet_replica.cc:330] T ae4bd63c4007403097cd0d9689c33676 P d03660fdd4aa4b61b2012134cdbf9425: stopping tablet replica
I20260810 22:29:01.696925 7689 raft_consensus.cc:2229] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:01.697065 7687 tablet_replica.cc:330] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a: stopping tablet replica
I20260810 22:29:01.697295 7689 raft_consensus.cc:2258] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:01.697646 7687 raft_consensus.cc:2229] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:01.697995 7687 raft_consensus.cc:2258] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:01.699194 7689 ts_tablet_manager.cc:1905] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:01.702935 7689 ts_tablet_manager.cc:1918] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:01.703068 7689 log.cc:1196] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/ae4bd63c4007403097cd0d9689c33676
I20260810 22:29:01.703385 7689 ts_tablet_manager.cc:1939] T ae4bd63c4007403097cd0d9689c33676 P c24de3341760467d97c2864931b16736: Deleting consensus metadata
I20260810 22:29:01.704605 7136 catalog_manager.cc:4841] TS c24de3341760467d97c2864931b16736 (127.1.221.130:36489): tablet ae4bd63c4007403097cd0d9689c33676 (table test-table [id=1a10d59ca57647dc8c531fc58357c429]) successfully deleted
I20260810 22:29:01.705377 7687 ts_tablet_manager.cc:1905] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 22:29:01.706854 7137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:41467 (error 108)
I20260810 22:29:01.707016 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 7359
I20260810 22:29:01.708333 7687 ts_tablet_manager.cc:1918] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:01.708449 7687 log.cc:1196] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/ae4bd63c4007403097cd0d9689c33676
I20260810 22:29:01.708793 7687 ts_tablet_manager.cc:1939] T ae4bd63c4007403097cd0d9689c33676 P e5efc4cbdbac49db80aa6c39a05f921a: Deleting consensus metadata
I20260810 22:29:01.710090 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.709694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41467 (local address 127.0.0.1:42310)
0810 22:29:01.709907 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:01.709991 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:41467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260810 22:29:01.710290 7137 catalog_manager.cc:4584] TS d03660fdd4aa4b61b2012134cdbf9425 (127.1.221.129:41467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae4bd63c4007403097cd0d9689c33676: Network error: Client connection negotiation failed: client connection to 127.1.221.129:41467: connect: Connection refused (error 111)
I20260810 22:29:01.710374 7138 catalog_manager.cc:4841] TS e5efc4cbdbac49db80aa6c39a05f921a (127.1.221.131:45213): tablet ae4bd63c4007403097cd0d9689c33676 (table test-table [id=1a10d59ca57647dc8c531fc58357c429]) successfully deleted
I20260810 22:29:01.715997 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 7504
I20260810 22:29:01.724536 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 7118
2026-08-10T22:29:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1109 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 22:29:01.748751 1910 mini_oidc.cc:152] Starting JWKS server
I20260810 22:29:01.750707 1910 webserver.cc:460] Webserver started at https://127.0.0.1:41215/ using document root <none> and password file <none>
I20260810 22:29:01.750917 1910 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 22:29:01.751214 1910 webserver.cc:460] Webserver started at http://127.0.0.1:45103/ using document root <none> and password file <none>
2026-08-10T22:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:01Z Disabled control of system clock
I20260810 22:29:01.761907 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34597
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:45499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:34597
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41215/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/testchainca.cert with env {}
W20260810 22:29:01.877046 7697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:01.877344 7697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:01.877386 7697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:01.881524 7697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:01.881628 7697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:01.881644 7697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:01.881661 7697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:01.881676 7697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 22:29:01.881692 7697 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 22:29:01.881706 7697 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41215/jwks.json
I20260810 22:29:01.887396 7697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:34597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:34597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41215/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:01.888506 7697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:01.889539 7697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:01.898546 7703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:01.898523 7705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:01.898523 7702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:01.898885 7697 server_base.cc:1034] running on GCE node
I20260810 22:29:01.899500 7697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:01.900959 7697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:01.902150 7697 hybrid_clock.cc:648] HybridClock initialized: now 1786400941902128 us; error 38 us; skew 500 ppm
I20260810 22:29:01.904134 7697 webserver.cc:460] Webserver started at http://127.1.221.190:40501/ using document root <none> and password file <none>
I20260810 22:29:01.904757 7697 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:01.904868 7697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:01.905092 7697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:01.907337 7697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "a66d17de972c41398043362a59214870"
format_stamp: "Formatted at 2026-08-10 22:29:01 on dist-test-slave-njxd"
server_key: "8309d8a4843ce315cadac92aaf0bd7b3"
server_key_iv: "b603413a68f478c9404ab517f47be92c"
server_key_version: "encryptionkey@0"
I20260810 22:29:01.907948 7697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "a66d17de972c41398043362a59214870"
format_stamp: "Formatted at 2026-08-10 22:29:01 on dist-test-slave-njxd"
server_key: "8309d8a4843ce315cadac92aaf0bd7b3"
server_key_iv: "b603413a68f478c9404ab517f47be92c"
server_key_version: "encryptionkey@0"
I20260810 22:29:01.911824 7697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:29:01.914417 7712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:01.915668 7697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:29:01.915822 7697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "a66d17de972c41398043362a59214870"
format_stamp: "Formatted at 2026-08-10 22:29:01 on dist-test-slave-njxd"
server_key: "8309d8a4843ce315cadac92aaf0bd7b3"
server_key_iv: "b603413a68f478c9404ab517f47be92c"
server_key_version: "encryptionkey@0"
I20260810 22:29:01.915943 7697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:01.936947 7697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:01.937842 7697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:01.938077 7697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:01.945369 7697 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:34597
I20260810 22:29:01.945398 7772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:34597 every 8 connection(s)
I20260810 22:29:01.946465 7697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:29:01.948516 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 7697
I20260810 22:29:01.948647 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:29:01.948975 1910 external_mini_cluster.cc:1428] Setting key a923f28eae16c93fe0f0e3008521fd99
I20260810 22:29:01.950716 7773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:01.956745 7773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870: Bootstrap starting.
I20260810 22:29:01.958719 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:01.950526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.1.221.190:34597)
0810 22:29:01.951280 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:01.951297 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:01.951337 (+ 40us) server_negotiation.cc:407] Connection header received
0810 22:29:01.952028 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:01.952045 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:01.952466 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:01.952838 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:01.953288 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:01.954359 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:01.955286 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:01.955780 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:01.955955 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:01.956906 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:01.956913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:01.956930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:01.957337 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:01.957394 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:01.957414 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:01.957526 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:01.957741 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 22:29:01.958002 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":156,"threads_started":1}
I20260810 22:29:01.959271 7773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:01.960110 7773 log.cc:824] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:01.961735 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:29:01.962321 7773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870: No bootstrap required, opened a new log
I20260810 22:29:01.965960 7773 raft_consensus.cc:348] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66d17de972c41398043362a59214870" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34597 } }
I20260810 22:29:01.966210 7773 raft_consensus.cc:374] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:01.966295 7773 raft_consensus.cc:729] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66d17de972c41398043362a59214870, State: Initialized, Role: FOLLOWER
I20260810 22:29:01.966724 7773 consensus_queue.cc:260] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66d17de972c41398043362a59214870" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34597 } }
I20260810 22:29:01.966883 7773 raft_consensus.cc:388] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:29:01.966969 7773 raft_consensus.cc:482] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:29:01.967088 7773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:01.968107 7773 raft_consensus.cc:504] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66d17de972c41398043362a59214870" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34597 } }
I20260810 22:29:01.968446 7773 leader_election.cc:302] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a66d17de972c41398043362a59214870; no voters:
I20260810 22:29:01.968874 7773 leader_election.cc:288] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:29:01.969017 7778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:01.969251 7778 raft_consensus.cc:686] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [term 1 LEADER]: Becoming Leader. State: Replica: a66d17de972c41398043362a59214870, State: Running, Role: LEADER
I20260810 22:29:01.969599 7778 consensus_queue.cc:237] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66d17de972c41398043362a59214870" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34597 } }
I20260810 22:29:01.970250 7773 sys_catalog.cc:564] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:01.971683 7781 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a66d17de972c41398043362a59214870. Latest consensus state: current_term: 1 leader_uuid: "a66d17de972c41398043362a59214870" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66d17de972c41398043362a59214870" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34597 } } }
I20260810 22:29:01.971658 7779 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a66d17de972c41398043362a59214870" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66d17de972c41398043362a59214870" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 34597 } } }
I20260810 22:29:01.971853 7781 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:01.971856 7779 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:01.972311 7791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:01.975841 7791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:01.982134 7791 catalog_manager.cc:1347] Generated new cluster ID: 219edace129f4f92b5591374ca1abf2b
I20260810 22:29:01.982203 7791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:01.991101 7791 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:29:01.992002 7791 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:02.005126 7791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a66d17de972c41398043362a59214870: Generated new TSK 0
I20260810 22:29:02.006040 7791 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 22:29:02.041342 7807 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 22:29:02.041477 7807 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 22:29:02.042186 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:02.024552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.1.221.190:34597)
0810 22:29:02.024699 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:02.024705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:02.024895 (+ 190us) server_negotiation.cc:407] Connection header received
0810 22:29:02.025094 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:02.025098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:02.025157 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:02.025273 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0810 22:29:02.025513 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.026383 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.027261 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.027642 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.027746 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:02.028064 (+ 318us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 22:29:02.041752 (+ 13688us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 22:29:02.042038 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54616: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 22:29:02.042273 7776 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54616: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 22:29:02.043524 7806 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:02.024430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:34597 (local address 127.0.0.1:54616)
0810 22:29:02.024811 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:02.024825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:02.024937 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:02.025299 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:02.025301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:02.025308 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0810 22:29:02.025406 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:02.025411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.026497 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.026502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:02.027145 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:02.027153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.027756 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.027759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:02.027830 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:02.027874 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 22:29:02.042007 (+ 14133us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 22:29:02.042214 (+ 207us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.221.190:34597: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":111,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.221.190:34597: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 22:29:02.044344 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 7697
2026-08-10T22:29:02Z chronyd exiting
I20260810 22:29:02.085233 1910 test_util.cc:175] -----------------------------------------------
I20260810 22:29:02.085320 1910 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786400913154531-1910-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (338 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 22:29:02.086547 1910 mini_oidc.cc:152] Starting JWKS server
I20260810 22:29:02.087445 1910 webserver.cc:460] Webserver started at https://127.0.0.1:36987/ using document root <none> and password file <none>
I20260810 22:29:02.087574 1910 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 22:29:02.087872 1910 webserver.cc:460] Webserver started at http://127.0.0.1:37729/ using document root <none> and password file <none>
2026-08-10T22:29:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:02Z Disabled control of system clock
I20260810 22:29:02.098388 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39653
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:39653
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36987/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/testchainca.cert with env {}
W20260810 22:29:02.212294 7814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:02.212594 7814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:02.212642 7814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:02.216670 7814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:02.216725 7814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:02.216740 7814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:02.216758 7814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:02.216773 7814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 22:29:02.216789 7814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 22:29:02.216801 7814 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36987/jwks.json
I20260810 22:29:02.222321 7814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:39653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36987/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:02.223409 7814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:02.224418 7814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:02.233080 7821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:02.233096 7824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:02.233099 7820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:02.233152 7814 server_base.cc:1034] running on GCE node
I20260810 22:29:02.233922 7814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:02.235060 7814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:02.236253 7814 hybrid_clock.cc:648] HybridClock initialized: now 1786400942236229 us; error 37 us; skew 500 ppm
I20260810 22:29:02.241986 7814 webserver.cc:460] Webserver started at http://127.1.221.190:42797/ using document root <none> and password file <none>
I20260810 22:29:02.242604 7814 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:02.242704 7814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:02.242949 7814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:02.245060 7814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "85870321b08c4a00a62de808d92f4535"
format_stamp: "Formatted at 2026-08-10 22:29:02 on dist-test-slave-njxd"
server_key: "fa082f1f2d554fe6fa312bd1e373a6ec"
server_key_iv: "d2ea74a0851f96bdfaa9bcf66141cbaf"
server_key_version: "encryptionkey@0"
I20260810 22:29:02.245725 7814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "85870321b08c4a00a62de808d92f4535"
format_stamp: "Formatted at 2026-08-10 22:29:02 on dist-test-slave-njxd"
server_key: "fa082f1f2d554fe6fa312bd1e373a6ec"
server_key_iv: "d2ea74a0851f96bdfaa9bcf66141cbaf"
server_key_version: "encryptionkey@0"
I20260810 22:29:02.249418 7814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:29:02.251942 7830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:02.253170 7814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:29:02.253335 7814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "85870321b08c4a00a62de808d92f4535"
format_stamp: "Formatted at 2026-08-10 22:29:02 on dist-test-slave-njxd"
server_key: "fa082f1f2d554fe6fa312bd1e373a6ec"
server_key_iv: "d2ea74a0851f96bdfaa9bcf66141cbaf"
server_key_version: "encryptionkey@0"
I20260810 22:29:02.253460 7814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:02.279554 7814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:02.280357 7814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:02.280579 7814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:02.287997 7814 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:39653
I20260810 22:29:02.288010 7887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:39653 every 8 connection(s)
I20260810 22:29:02.289075 7814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:29:02.292357 7888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:02.295212 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 7814
I20260810 22:29:02.295351 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:29:02.295631 1910 external_mini_cluster.cc:1428] Setting key d0220535077f65ccd01b01fbc9598cc6
I20260810 22:29:02.299639 7888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535: Bootstrap starting.
I20260810 22:29:02.302273 7888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:02.303272 7888 log.cc:824] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:02.305486 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:02.297124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.1.221.190:39653)
0810 22:29:02.297823 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:02.297841 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:02.297880 (+ 39us) server_negotiation.cc:407] Connection header received
0810 22:29:02.298651 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:02.298671 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:02.299097 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:02.299392 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:02.299825 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.300804 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.301991 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.302521 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.302641 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:02.303537 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:02.303543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:02.303556 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:02.304092 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:02.304177 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:02.304199 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:02.304289 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:02.304452 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 22:29:02.304752 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":121,"threads_started":1}
I20260810 22:29:02.305495 7888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535: No bootstrap required, opened a new log
I20260810 22:29:02.308590 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:29:02.310149 7888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85870321b08c4a00a62de808d92f4535" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39653 } }
I20260810 22:29:02.310451 7888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:02.310521 7888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85870321b08c4a00a62de808d92f4535, State: Initialized, Role: FOLLOWER
I20260810 22:29:02.310943 7888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85870321b08c4a00a62de808d92f4535" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39653 } }
I20260810 22:29:02.311105 7888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:29:02.311175 7888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:29:02.311280 7888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:02.312304 7888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85870321b08c4a00a62de808d92f4535" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39653 } }
I20260810 22:29:02.312654 7888 leader_election.cc:302] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85870321b08c4a00a62de808d92f4535; no voters:
I20260810 22:29:02.312991 7888 leader_election.cc:288] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:29:02.313159 7894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:02.313392 7894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [term 1 LEADER]: Becoming Leader. State: Replica: 85870321b08c4a00a62de808d92f4535, State: Running, Role: LEADER
I20260810 22:29:02.313730 7894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85870321b08c4a00a62de808d92f4535" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39653 } }
I20260810 22:29:02.314313 7888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:02.315562 7896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85870321b08c4a00a62de808d92f4535. Latest consensus state: current_term: 1 leader_uuid: "85870321b08c4a00a62de808d92f4535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85870321b08c4a00a62de808d92f4535" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39653 } } }
I20260810 22:29:02.315704 7896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:02.315948 7895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85870321b08c4a00a62de808d92f4535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85870321b08c4a00a62de808d92f4535" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39653 } } }
I20260810 22:29:02.316040 7895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:02.316316 7904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:02.319550 7904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:02.325491 7904 catalog_manager.cc:1347] Generated new cluster ID: ffedea61c0a34b05a31c67364d7adfda
I20260810 22:29:02.325599 7904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:02.351905 7904 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:29:02.353772 7904 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:02.365706 7904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85870321b08c4a00a62de808d92f4535: Generated new TSK 0
I20260810 22:29:02.366598 7904 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 22:29:02.400374 7924 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 22:29:02.400485 7924 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 22:29:02.401268 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:02.385034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38264 (local address 127.1.221.190:39653)
0810 22:29:02.385188 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:02.385193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:02.385338 (+ 145us) server_negotiation.cc:407] Connection header received
0810 22:29:02.385501 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:02.385505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:02.385565 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:02.385714 (+ 149us) server_negotiation.cc:226] Negotiated authn=JWT
0810 22:29:02.385936 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.386790 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.387682 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.388056 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.388163 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:02.388454 (+ 291us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 22:29:02.400793 (+ 12339us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 22:29:02.401096 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38264: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 22:29:02.401343 7891 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38264: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 22:29:02.401378 7923 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:02.384860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39653 (local address 127.0.0.1:38264)
0810 22:29:02.385245 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:02.385258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:02.385358 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:02.385719 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:02.385721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:02.385728 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0810 22:29:02.385823 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:02.385828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.386908 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.386911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:02.387562 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:02.387569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.388179 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.388182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:02.388247 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:02.388271 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 22:29:02.401021 (+ 12750us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 22:29:02.401196 (+ 175us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.221.190:39653: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":113,"threads_started":1}
I20260810 22:29:02.401844 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 7814
2026-08-10T22:29:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (360 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 22:29:02.447479 1910 mini_oidc.cc:152] Starting JWKS server
I20260810 22:29:02.448412 1910 webserver.cc:460] Webserver started at https://127.0.0.1:41815/ using document root <none> and password file <none>
I20260810 22:29:02.448534 1910 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 22:29:02.448832 1910 webserver.cc:460] Webserver started at http://127.0.0.1:40273/ using document root <none> and password file <none>
2026-08-10T22:29:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:02Z Disabled control of system clock
I20260810 22:29:02.459374 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44015
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:44015
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41815/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 22:29:02.573968 7931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:02.574326 7931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:02.574410 7931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:02.578684 7931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:02.578796 7931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:02.578828 7931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:02.578869 7931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:02.578903 7931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 22:29:02.578936 7931 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 22:29:02.578967 7931 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41815/jwks.json
I20260810 22:29:02.583901 7931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:44015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41815/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:02.585129 7931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:02.586639 7931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:02.595506 7937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:02.595486 7938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:02.595522 7942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:02.595590 7931 server_base.cc:1034] running on GCE node
I20260810 22:29:02.596405 7931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:02.597631 7931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:02.598840 7931 hybrid_clock.cc:648] HybridClock initialized: now 1786400942598820 us; error 34 us; skew 500 ppm
I20260810 22:29:02.600905 7931 webserver.cc:460] Webserver started at http://127.1.221.190:42737/ using document root <none> and password file <none>
I20260810 22:29:02.601531 7931 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:02.601658 7931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:02.601977 7931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:02.604199 7931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "953cd349f5e3407fbb0aec850a0fab88"
format_stamp: "Formatted at 2026-08-10 22:29:02 on dist-test-slave-njxd"
server_key: "cecb2667b4803c9a62d2e3def2fe2beb"
server_key_iv: "c3eb7e8189385e57a6d9dea148ef9da6"
server_key_version: "encryptionkey@0"
I20260810 22:29:02.604863 7931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "953cd349f5e3407fbb0aec850a0fab88"
format_stamp: "Formatted at 2026-08-10 22:29:02 on dist-test-slave-njxd"
server_key: "cecb2667b4803c9a62d2e3def2fe2beb"
server_key_iv: "c3eb7e8189385e57a6d9dea148ef9da6"
server_key_version: "encryptionkey@0"
I20260810 22:29:02.608889 7931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 22:29:02.611497 7949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:02.612761 7931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:29:02.612914 7931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "953cd349f5e3407fbb0aec850a0fab88"
format_stamp: "Formatted at 2026-08-10 22:29:02 on dist-test-slave-njxd"
server_key: "cecb2667b4803c9a62d2e3def2fe2beb"
server_key_iv: "c3eb7e8189385e57a6d9dea148ef9da6"
server_key_version: "encryptionkey@0"
I20260810 22:29:02.613036 7931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:02.643610 7931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:02.644439 7931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:02.644661 7931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:02.652256 7931 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:44015
I20260810 22:29:02.652264 8005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:44015 every 8 connection(s)
I20260810 22:29:02.653260 7931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:29:02.656035 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 7931
I20260810 22:29:02.656194 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:29:02.656518 1910 external_mini_cluster.cc:1428] Setting key e4e10c4d9eaa16b048f8c9f4d8d401c1
I20260810 22:29:02.656534 8006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:02.664230 8006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88: Bootstrap starting.
I20260810 22:29:02.667138 8006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:02.667475 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:02.658101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.1.221.190:44015)
0810 22:29:02.658824 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:02.658844 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:02.658892 (+ 48us) server_negotiation.cc:407] Connection header received
0810 22:29:02.659575 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:02.659600 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:02.660147 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:02.660550 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:02.661026 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.662487 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.663429 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.664122 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.664289 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:02.665203 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:02.665209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:02.665226 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:02.665818 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:02.665890 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:02.665917 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:02.666014 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:02.666173 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 22:29:02.666448 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":128,"threads_started":1}
I20260810 22:29:02.668022 8006 log.cc:824] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:02.670574 8006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88: No bootstrap required, opened a new log
I20260810 22:29:02.670650 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:29:02.674816 8006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953cd349f5e3407fbb0aec850a0fab88" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44015 } }
I20260810 22:29:02.675153 8006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:02.675226 8006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 953cd349f5e3407fbb0aec850a0fab88, State: Initialized, Role: FOLLOWER
I20260810 22:29:02.675736 8006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953cd349f5e3407fbb0aec850a0fab88" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44015 } }
I20260810 22:29:02.675915 8006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:29:02.675984 8006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:29:02.676085 8006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:02.677165 8006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953cd349f5e3407fbb0aec850a0fab88" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44015 } }
I20260810 22:29:02.677528 8006 leader_election.cc:302] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 953cd349f5e3407fbb0aec850a0fab88; no voters:
I20260810 22:29:02.677953 8006 leader_election.cc:288] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:29:02.678159 8012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:02.678431 8012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [term 1 LEADER]: Becoming Leader. State: Replica: 953cd349f5e3407fbb0aec850a0fab88, State: Running, Role: LEADER
I20260810 22:29:02.678766 8012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953cd349f5e3407fbb0aec850a0fab88" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44015 } }
I20260810 22:29:02.679349 8006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:02.680580 8013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "953cd349f5e3407fbb0aec850a0fab88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953cd349f5e3407fbb0aec850a0fab88" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44015 } } }
I20260810 22:29:02.680727 8013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:02.681067 8014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 953cd349f5e3407fbb0aec850a0fab88. Latest consensus state: current_term: 1 leader_uuid: "953cd349f5e3407fbb0aec850a0fab88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953cd349f5e3407fbb0aec850a0fab88" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44015 } } }
I20260810 22:29:02.681164 8014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88 [sys.catalog]: This master's current role is: LEADER
W20260810 22:29:02.684337 8030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:29:02.684446 8030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:29:02.684525 8022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:02.685267 8022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:02.690785 8022 catalog_manager.cc:1347] Generated new cluster ID: d5a369a4939748fab7cec0ed6c2bba5c
I20260810 22:29:02.690865 8022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:02.700405 8022 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:29:02.701345 8022 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:02.722123 8022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 953cd349f5e3407fbb0aec850a0fab88: Generated new TSK 0
I20260810 22:29:02.722908 8022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:29:02.742357 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:02.735303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.1.221.190:44015)
0810 22:29:02.735454 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:02.735459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:02.735672 (+ 213us) server_negotiation.cc:407] Connection header received
0810 22:29:02.735860 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:02.735863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:02.735910 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:02.736002 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0810 22:29:02.736278 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.736816 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.738110 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.738328 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.738407 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:02.738746 (+ 339us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 22:29:02.741965 (+ 3219us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41815/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 22:29:02.742202 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41815/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 22:29:02.742430 8010 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43886: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41815/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 22:29:02.744155 8039 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:02.735220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44015 (local address 127.0.0.1:43886)
0810 22:29:02.735569 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:02.735586 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:02.735709 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:02.736030 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:02.736033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:02.736041 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 22:29:02.736169 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:02.736176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.736931 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.736934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:02.737979 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:02.737990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.738452 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.738456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:02.738547 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:02.738576 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 22:29:02.742147 (+ 3571us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41815/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 22:29:02.743954 (+ 1807us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.221.190:44015: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41815/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":1566,"thread_start_us":137,"threads_started":1}
I20260810 22:29:02.744764 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 7931
E20260810 22:29:02.753650 8040 webserver.cc:533] Webserver: sslize failed:
E20260810 22:29:02.753757 8040 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T22:29:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (338 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T22:29:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:02Z Disabled control of system clock
I20260810 22:29:02.796831 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39261
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:39905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:39261
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/testchainca.cert with env {}
W20260810 22:29:02.912029 8045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:02.912319 8045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:02.912361 8045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:02.916378 8045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:02.916438 8045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:02.916455 8045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:02.916471 8045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:02.916486 8045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 22:29:02.916501 8045 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 22:29:02.916514 8045 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 22:29:02.921921 8045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:39905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:39261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:02.923070 8045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:02.924079 8045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:29:02.932830 8045 server_base.cc:1034] running on GCE node
W20260810 22:29:02.932849 8051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:02.932773 8050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:02.932773 8053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:02.933557 8045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:02.934836 8045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:02.935999 8045 hybrid_clock.cc:648] HybridClock initialized: now 1786400942935969 us; error 44 us; skew 500 ppm
I20260810 22:29:02.938058 8045 webserver.cc:460] Webserver started at http://127.1.221.190:41093/ using document root <none> and password file <none>
I20260810 22:29:02.938663 8045 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:02.938730 8045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:02.938982 8045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:02.941217 8045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "0c303bad7ac74dc5999f69221fafcd8a"
format_stamp: "Formatted at 2026-08-10 22:29:02 on dist-test-slave-njxd"
server_key: "91c91fffa5a50cc7e151c15e225b85a6"
server_key_iv: "817e27f3fabe9d11ffd40391fbad5748"
server_key_version: "encryptionkey@0"
I20260810 22:29:02.941896 8045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "0c303bad7ac74dc5999f69221fafcd8a"
format_stamp: "Formatted at 2026-08-10 22:29:02 on dist-test-slave-njxd"
server_key: "91c91fffa5a50cc7e151c15e225b85a6"
server_key_iv: "817e27f3fabe9d11ffd40391fbad5748"
server_key_version: "encryptionkey@0"
I20260810 22:29:02.945986 8045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:29:02.948740 8059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:02.950121 8045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:29:02.950253 8045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "0c303bad7ac74dc5999f69221fafcd8a"
format_stamp: "Formatted at 2026-08-10 22:29:02 on dist-test-slave-njxd"
server_key: "91c91fffa5a50cc7e151c15e225b85a6"
server_key_iv: "817e27f3fabe9d11ffd40391fbad5748"
server_key_version: "encryptionkey@0"
I20260810 22:29:02.950363 8045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:02.966598 8045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:02.967402 8045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:02.967621 8045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:02.975334 8045 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:39261
I20260810 22:29:02.975339 8118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:39261 every 8 connection(s)
I20260810 22:29:02.976341 8045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:29:02.979741 8119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:02.983409 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 8045
I20260810 22:29:02.983569 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:29:02.983860 1910 external_mini_cluster.cc:1428] Setting key bbe335d58f8f26edcb7beb740871af8c
I20260810 22:29:02.985922 8119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a: Bootstrap starting.
I20260810 22:29:02.988335 8119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:02.989116 8119 log.cc:824] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:02.991734 8119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a: No bootstrap required, opened a new log
I20260810 22:29:02.994431 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:02.985279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.1.221.190:39261)
0810 22:29:02.986015 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:02.986057 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:02.986108 (+ 51us) server_negotiation.cc:407] Connection header received
0810 22:29:02.986748 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:02.986765 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:02.987161 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:02.987507 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:02.987960 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.989263 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.990533 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:02.991226 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:02.991397 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:02.992295 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:02.992301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:02.992317 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:02.992811 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:02.992890 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:02.992915 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:02.993016 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:02.993155 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 22:29:02.993437 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":139,"threads_started":1}
I20260810 22:29:02.995537 8119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c303bad7ac74dc5999f69221fafcd8a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39261 } }
I20260810 22:29:02.995796 8119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:02.995878 8119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c303bad7ac74dc5999f69221fafcd8a, State: Initialized, Role: FOLLOWER
I20260810 22:29:02.996342 8119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c303bad7ac74dc5999f69221fafcd8a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39261 } }
I20260810 22:29:02.996487 8119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:29:02.996542 8119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:29:02.996625 8119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:02.997841 8119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c303bad7ac74dc5999f69221fafcd8a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39261 } }
I20260810 22:29:02.997991 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:29:02.998227 8119 leader_election.cc:302] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c303bad7ac74dc5999f69221fafcd8a; no voters:
I20260810 22:29:02.998551 8119 leader_election.cc:288] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:29:02.998687 8125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:02.998919 8125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [term 1 LEADER]: Becoming Leader. State: Replica: 0c303bad7ac74dc5999f69221fafcd8a, State: Running, Role: LEADER
I20260810 22:29:02.999228 8125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c303bad7ac74dc5999f69221fafcd8a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39261 } }
I20260810 22:29:02.999893 8119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:03.000898 8126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c303bad7ac74dc5999f69221fafcd8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c303bad7ac74dc5999f69221fafcd8a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39261 } } }
I20260810 22:29:03.001050 8126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:03.001350 8128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c303bad7ac74dc5999f69221fafcd8a. Latest consensus state: current_term: 1 leader_uuid: "0c303bad7ac74dc5999f69221fafcd8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c303bad7ac74dc5999f69221fafcd8a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39261 } } }
I20260810 22:29:03.001430 8128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:03.001431 8136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:03.004238 8136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:03.010190 8136 catalog_manager.cc:1347] Generated new cluster ID: 27d5d055f5ea40818fa88d9d8d6000fe
I20260810 22:29:03.010262 8136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:03.023311 8136 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:29:03.024474 8136 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:03.034394 8136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c303bad7ac74dc5999f69221fafcd8a: Generated new TSK 0
I20260810 22:29:03.035243 8136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:29:03.085139 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:03.048721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.1.221.190:39261)
0810 22:29:03.048875 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:03.048880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:03.049046 (+ 166us) server_negotiation.cc:407] Connection header received
0810 22:29:03.049192 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:03.049195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:03.049263 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:03.049390 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0810 22:29:03.049639 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:03.050464 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:03.051346 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:03.051718 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:03.051822 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:03.052135 (+ 313us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 22:29:03.084623 (+ 32488us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 22:29:03.084976 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53830: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 22:29:03.085239 8123 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53830: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 22:29:03.085296 8153 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:03.048573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39261 (local address 127.0.0.1:53830)
0810 22:29:03.048951 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:03.048965 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:03.049072 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:03.049394 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:03.049396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:03.049403 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0810 22:29:03.049504 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:03.049508 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:03.050590 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:03.050593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:03.051223 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:03.051231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:03.051847 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:03.051849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:03.051917 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:03.051942 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 22:29:03.084884 (+ 32942us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 22:29:03.085094 (+ 210us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.221.190:39261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":125,"threads_started":1}
I20260810 22:29:03.086383 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 8045
2026-08-10T22:29:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (322 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (130 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (130 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:29:03 dist-test-slave-njxd krb5kdc[8166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:29:03 dist-test-slave-njxd krb5kdc[8166](info): set up 2 sockets
Aug 10 22:29:03 dist-test-slave-njxd krb5kdc[8166](info): commencing operation
krb5kdc: starting...
W20260810 22:29:04.443446 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:29:04 dist-test-slave-njxd krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:29:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:29:04.568713 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43679
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:43679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:04.684736 8179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:04.685043 8179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:04.685127 8179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:04.689170 8179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:04.689265 8179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:04.689306 8179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:04.689363 8179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:04.689404 8179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:29:04.694816 8179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:43679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:04.695925 8179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:04.697005 8179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:04.705943 8185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:04.705946 8187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:04.706018 8179 server_base.cc:1034] running on GCE node
W20260810 22:29:04.705951 8184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:04.706770 8179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:04.707963 8179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:04.709156 8179 hybrid_clock.cc:648] HybridClock initialized: now 1786400944709125 us; error 41 us; skew 500 ppm
Aug 10 22:29:04 dist-test-slave-njxd krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400944, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:04.711993 8179 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:29:04.712944 8179 webserver.cc:460] Webserver started at http://127.1.221.190:39209/ using document root <none> and password file <none>
I20260810 22:29:04.713519 8179 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:04.713649 8179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:04.713893 8179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:04.716007 8179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "e8f2e6be27c14c869c4b231879d61990"
format_stamp: "Formatted at 2026-08-10 22:29:04 on dist-test-slave-njxd"
server_key: "c688dcf1104a216fc03203682b75353d"
server_key_iv: "77ba7519bdde903135f9280f7a932dc3"
server_key_version: "encryptionkey@0"
I20260810 22:29:04.716655 8179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "e8f2e6be27c14c869c4b231879d61990"
format_stamp: "Formatted at 2026-08-10 22:29:04 on dist-test-slave-njxd"
server_key: "c688dcf1104a216fc03203682b75353d"
server_key_iv: "77ba7519bdde903135f9280f7a932dc3"
server_key_version: "encryptionkey@0"
I20260810 22:29:04.720440 8179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 22:29:04.722892 8195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:04.724082 8179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:29:04.724222 8179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "e8f2e6be27c14c869c4b231879d61990"
format_stamp: "Formatted at 2026-08-10 22:29:04 on dist-test-slave-njxd"
server_key: "c688dcf1104a216fc03203682b75353d"
server_key_iv: "77ba7519bdde903135f9280f7a932dc3"
server_key_version: "encryptionkey@0"
I20260810 22:29:04.724335 8179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:04.741787 8179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:04.744668 8179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:04.744886 8179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:04.752656 8179 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:43679
I20260810 22:29:04.752683 8252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:43679 every 8 connection(s)
I20260810 22:29:04.753669 8179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:29:04.755867 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 8179
I20260810 22:29:04.755985 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:29:04.756297 1910 external_mini_cluster.cc:1428] Setting key eca2f6db3a600b45ea182942015f1f17
I20260810 22:29:04.756582 8253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:04.763055 8253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990: Bootstrap starting.
Aug 10 22:29:04 dist-test-slave-njxd krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:29:04.765779 8253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:04.766623 8253 log.cc:824] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:04.769068 8253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990: No bootstrap required, opened a new log
I20260810 22:29:04.771046 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:04.757789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39622 (local address 127.1.221.190:43679)
0810 22:29:04.758313 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:04.758329 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:04.758365 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:29:04.759021 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:04.759038 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:04.759375 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:04.759715 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:04.760586 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:04.761647 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:04.762519 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:04.763044 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:04.763240 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:04.766047 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:04.766053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:04.766067 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:04.766097 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:04.768039 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:04.768579 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:04.768584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:04.768589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:04.768716 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:04.769024 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:04.769026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:04.769027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:04.769503 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:04.769725 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:04.769986 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 22:29:04.770298 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":172,"threads_started":1}
I20260810 22:29:04.773231 8253 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f2e6be27c14c869c4b231879d61990" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43679 } }
I20260810 22:29:04.773494 8253 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:04.773574 8253 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f2e6be27c14c869c4b231879d61990, State: Initialized, Role: FOLLOWER
I20260810 22:29:04.774027 8253 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f2e6be27c14c869c4b231879d61990" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43679 } }
I20260810 22:29:04.774184 8253 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:29:04.774271 8253 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:29:04.774394 8253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:04.775350 8253 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f2e6be27c14c869c4b231879d61990" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43679 } }
I20260810 22:29:04.775694 8253 leader_election.cc:302] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f2e6be27c14c869c4b231879d61990; no voters:
I20260810 22:29:04.776010 8253 leader_election.cc:288] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:29:04.776154 8260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:04.776433 8260 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [term 1 LEADER]: Becoming Leader. State: Replica: e8f2e6be27c14c869c4b231879d61990, State: Running, Role: LEADER
I20260810 22:29:04.776821 8260 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f2e6be27c14c869c4b231879d61990" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43679 } }
I20260810 22:29:04.777405 8253 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:04.778726 8262 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8f2e6be27c14c869c4b231879d61990. Latest consensus state: current_term: 1 leader_uuid: "e8f2e6be27c14c869c4b231879d61990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f2e6be27c14c869c4b231879d61990" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43679 } } }
I20260810 22:29:04.778751 8261 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8f2e6be27c14c869c4b231879d61990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f2e6be27c14c869c4b231879d61990" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43679 } } }
I20260810 22:29:04.778905 8262 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:04.778905 8261 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990 [sys.catalog]: This master's current role is: LEADER
W20260810 22:29:04.782650 8276 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:29:04.782738 8276 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:29:04.782800 8271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:04.783679 8271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:04.789563 8271 catalog_manager.cc:1347] Generated new cluster ID: 4c0d282b531d4a4a8f70de0994c8f35f
I20260810 22:29:04.789698 8271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:04.808547 8271 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:29:04.809505 8271 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:04.816115 8271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8f2e6be27c14c869c4b231879d61990: Generated new TSK 0
I20260810 22:29:04.816807 8271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:29:04.854565 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43679
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:04.972071 8282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:04.972343 8282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:04.972401 8282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:04.976406 8282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:04.976478 8282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:04.976537 8282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:29:04.982359 8282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:04.983492 8282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:04.984535 8282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:04.994436 8287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:04.994590 8289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:04.994690 8291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:04.994521 8282 server_base.cc:1034] running on GCE node
I20260810 22:29:04.995101 8282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:04.995741 8282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:04.996934 8282 hybrid_clock.cc:648] HybridClock initialized: now 1786400944996896 us; error 50 us; skew 500 ppm
Aug 10 22:29:04 dist-test-slave-njxd krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400944, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:04.999673 8282 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:29:05.000505 8282 webserver.cc:460] Webserver started at http://127.1.221.129:44441/ using document root <none> and password file <none>
I20260810 22:29:05.001086 8282 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:05.001194 8282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:05.001406 8282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:05.003489 8282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "fd0fa52076074ff49c7053f186e6ecd2"
format_stamp: "Formatted at 2026-08-10 22:29:04 on dist-test-slave-njxd"
server_key: "464cf9fa9e830145b99d22b440a6b9c3"
server_key_iv: "985c8d215a60be39a3244840da79e6f2"
server_key_version: "encryptionkey@0"
I20260810 22:29:05.004010 8282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "fd0fa52076074ff49c7053f186e6ecd2"
format_stamp: "Formatted at 2026-08-10 22:29:04 on dist-test-slave-njxd"
server_key: "464cf9fa9e830145b99d22b440a6b9c3"
server_key_iv: "985c8d215a60be39a3244840da79e6f2"
server_key_version: "encryptionkey@0"
I20260810 22:29:05.007751 8282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:29:05.010223 8300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:05.011541 8282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:29:05.011659 8282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "fd0fa52076074ff49c7053f186e6ecd2"
format_stamp: "Formatted at 2026-08-10 22:29:04 on dist-test-slave-njxd"
server_key: "464cf9fa9e830145b99d22b440a6b9c3"
server_key_iv: "985c8d215a60be39a3244840da79e6f2"
server_key_version: "encryptionkey@0"
I20260810 22:29:05.011765 8282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:05.031425 8282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:05.034571 8282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:05.034789 8282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:05.035427 8282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:05.036309 8282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:05.036355 8282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:05.036427 8282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:05.036456 8282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:05.045686 8282 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:35001
I20260810 22:29:05.045706 8420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:35001 every 8 connection(s)
I20260810 22:29:05.046646 8282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:29:05.052199 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 8282
I20260810 22:29:05.052340 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:29:05.052614 1910 external_mini_cluster.cc:1428] Setting key 6c66d3d0b4a92b6f93b7089e6a8c93e9
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400944, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:29:05.061177 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:05.048381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:54295 (local address 127.1.221.190:43679)
0810 22:29:05.048518 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:05.048521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:05.049337 (+ 816us) server_negotiation.cc:407] Connection header received
0810 22:29:05.050763 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:05.050766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:05.050823 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:05.050900 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:05.052081 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.052801 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.054200 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.054567 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.054713 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:05.057210 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:05.057213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:05.057215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:05.057238 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:05.058826 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:05.059533 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:05.059536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:05.059537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:05.059613 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:05.060003 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:05.060005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:05.060006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:05.060172 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:05.060247 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:05.060944 (+ 697us) server_negotiation.cc:299] Negotiation successful
0810 22:29:05.061052 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:29:05.062476 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:05.048699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43679 (local address 127.1.221.129:54295)
0810 22:29:05.049167 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:05.049209 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:05.050533 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:05.051033 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:05.051040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:05.051398 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:05.051877 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:05.051890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.052933 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.052936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:05.054050 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:05.054062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.054750 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.054753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:05.054893 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:05.055515 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:05.055531 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:05.057031 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:05.058956 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:05.058966 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:05.058985 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:05.059407 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:05.059717 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:05.059720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:05.059722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:05.059907 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:05.060290 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:05.060300 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:05.060642 (+ 342us) client_negotiation.cc:769] Sending connection context
0810 22:29:05.060915 (+ 273us) client_negotiation.cc:240] Negotiation successful
0810 22:29:05.061312 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":127,"threads_started":1}
I20260810 22:29:05.063795 8421 heartbeater.cc:347] Connected to a master server at 127.1.221.190:43679
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260810 22:29:05.064073 8421 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:05.064610 8421 heartbeater.cc:511] Master 127.1.221.190:43679 requested a full tablet report, sending...
I20260810 22:29:05.066206 8214 ts_manager.cc:194] Registered new tserver with Master: fd0fa52076074ff49c7053f186e6ecd2 (127.1.221.129:35001)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260810 22:29:05.067493 8214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:54295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:29:05.085443 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43679
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:05.201411 8426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:05.201717 8426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:05.201805 8426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:05.205540 8426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:05.205655 8426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:05.205745 8426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:29:05.210857 8426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:05.211932 8426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:05.212999 8426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:05.223161 8433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:05.223217 8426 server_base.cc:1034] running on GCE node
W20260810 22:29:05.223237 8436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:05.223172 8431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:05.223868 8426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:05.224480 8426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:05.225698 8426 hybrid_clock.cc:648] HybridClock initialized: now 1786400945225676 us; error 29 us; skew 500 ppm
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:05.228606 8426 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:29:05.229674 8426 webserver.cc:460] Webserver started at http://127.1.221.130:40065/ using document root <none> and password file <none>
I20260810 22:29:05.230253 8426 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:05.230340 8426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:05.230588 8426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:05.232730 8426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "d5e46f192a26479b9cde12c1ac631695"
format_stamp: "Formatted at 2026-08-10 22:29:05 on dist-test-slave-njxd"
server_key: "c0b5d20f725a00b853dc3e86761292d0"
server_key_iv: "798fe4ac1716b0c34e8ec130eacac82b"
server_key_version: "encryptionkey@0"
I20260810 22:29:05.233315 8426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "d5e46f192a26479b9cde12c1ac631695"
format_stamp: "Formatted at 2026-08-10 22:29:05 on dist-test-slave-njxd"
server_key: "c0b5d20f725a00b853dc3e86761292d0"
server_key_iv: "798fe4ac1716b0c34e8ec130eacac82b"
server_key_version: "encryptionkey@0"
I20260810 22:29:05.237160 8426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:29:05.239537 8446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:05.240810 8426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:29:05.240933 8426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "d5e46f192a26479b9cde12c1ac631695"
format_stamp: "Formatted at 2026-08-10 22:29:05 on dist-test-slave-njxd"
server_key: "c0b5d20f725a00b853dc3e86761292d0"
server_key_iv: "798fe4ac1716b0c34e8ec130eacac82b"
server_key_version: "encryptionkey@0"
I20260810 22:29:05.241034 8426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:05.267406 8426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:05.270975 8426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:05.271212 8426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:05.271862 8426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:05.272775 8426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:05.272828 8426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:05.272914 8426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:05.272950 8426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:05.283720 8426 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:33621
I20260810 22:29:05.283739 8570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:33621 every 8 connection(s)
I20260810 22:29:05.284662 8426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:29:05.293398 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 8426
I20260810 22:29:05.293540 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:29:05.293854 1910 external_mini_cluster.cc:1428] Setting key ea9ff82558702a9279f614ac5c38b8fa
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:29:05.299008 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:05.286503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:47387 (local address 127.1.221.190:43679)
0810 22:29:05.286645 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:05.286649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:05.287517 (+ 868us) server_negotiation.cc:407] Connection header received
0810 22:29:05.289004 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:05.289007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:05.289069 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:05.289170 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:05.290596 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.291133 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.292288 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.292515 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.292631 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:05.295498 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:05.295502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:05.295510 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:05.295534 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:05.296889 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:05.297438 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:05.297441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:05.297442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:05.297517 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:05.297892 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:05.297897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:05.297898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:05.298076 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:05.298190 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:05.298690 (+ 500us) server_negotiation.cc:299] Negotiation successful
0810 22:29:05.298826 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:29:05.299803 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:05.286917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43679 (local address 127.1.221.130:47387)
0810 22:29:05.287363 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:05.287408 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:05.288771 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:05.289358 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:05.289365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:05.289819 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:05.290426 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:05.290440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.291281 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.291285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:05.292156 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:05.292165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.292675 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.292679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:05.292848 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:05.293567 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:05.293636 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:05.295340 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:05.297017 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:05.297023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:05.297038 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:05.297303 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:05.297639 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:05.297643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:05.297644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:05.297792 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:05.298190 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:05.298197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:05.298446 (+ 249us) client_negotiation.cc:769] Sending connection context
0810 22:29:05.298632 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 22:29:05.298955 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":120,"threads_started":1}
I20260810 22:29:05.300943 8571 heartbeater.cc:347] Connected to a master server at 127.1.221.190:43679
I20260810 22:29:05.301229 8571 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:05.301812 8571 heartbeater.cc:511] Master 127.1.221.190:43679 requested a full tablet report, sending...
I20260810 22:29:05.302816 8214 ts_manager.cc:194] Registered new tserver with Master: d5e46f192a26479b9cde12c1ac631695 (127.1.221.130:33621)
I20260810 22:29:05.303411 8214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:47387
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:29:05.325120 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43679
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:05.441006 8578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:05.441380 8578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:05.441434 8578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:05.446988 8578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:05.447077 8578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:05.447171 8578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:29:05.452850 8578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:05.454052 8578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:05.455152 8578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:05.465126 8583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:05.465129 8584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:05.465200 8578 server_base.cc:1034] running on GCE node
W20260810 22:29:05.465126 8586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:05.465880 8578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:05.466483 8578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:05.467686 8578 hybrid_clock.cc:648] HybridClock initialized: now 1786400945467664 us; error 25 us; skew 500 ppm
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:05.470582 8578 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:29:05.471474 8578 webserver.cc:460] Webserver started at http://127.1.221.131:35013/ using document root <none> and password file <none>
I20260810 22:29:05.472067 8578 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:05.472164 8578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:05.472406 8578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:05.474586 8578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "a892cd1837b940c99eb4e804dc373034"
format_stamp: "Formatted at 2026-08-10 22:29:05 on dist-test-slave-njxd"
server_key: "7b4011b319707c366f0a45a6643dcc25"
server_key_iv: "994d483f37db3fb381c647ef3adcd93b"
server_key_version: "encryptionkey@0"
I20260810 22:29:05.475188 8578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "a892cd1837b940c99eb4e804dc373034"
format_stamp: "Formatted at 2026-08-10 22:29:05 on dist-test-slave-njxd"
server_key: "7b4011b319707c366f0a45a6643dcc25"
server_key_iv: "994d483f37db3fb381c647ef3adcd93b"
server_key_version: "encryptionkey@0"
I20260810 22:29:05.479435 8578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:29:05.482065 8594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:05.483385 8578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:29:05.483531 8578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "a892cd1837b940c99eb4e804dc373034"
format_stamp: "Formatted at 2026-08-10 22:29:05 on dist-test-slave-njxd"
server_key: "7b4011b319707c366f0a45a6643dcc25"
server_key_iv: "994d483f37db3fb381c647ef3adcd93b"
server_key_version: "encryptionkey@0"
I20260810 22:29:05.483656 8578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:05.496892 8578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:05.499970 8578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:05.500172 8578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:05.500806 8578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:05.501667 8578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:05.501748 8578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:05.501824 8578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:05.501869 8578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:05.510680 8578 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:36655
I20260810 22:29:05.510694 8722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:36655 every 8 connection(s)
I20260810 22:29:05.511576 8578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:29:05.512506 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 8578
I20260810 22:29:05.512627 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:29:05.512923 1910 external_mini_cluster.cc:1428] Setting key 516a3b99335a561c45206f8c4e17e60f
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:29:05.525604 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:05.513521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:33303 (local address 127.1.221.190:43679)
0810 22:29:05.513679 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:05.513683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:05.514577 (+ 894us) server_negotiation.cc:407] Connection header received
0810 22:29:05.516228 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:05.516231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:05.516286 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:05.516383 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:05.517562 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.518138 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.519230 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.519442 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.519519 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:05.521966 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:05.521970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:05.521973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:05.522004 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:05.523467 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:05.524002 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:05.524006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:05.524008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:05.524101 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:05.524475 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:05.524479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:05.524481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:05.524675 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:05.524786 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:05.525284 (+ 498us) server_negotiation.cc:299] Negotiation successful
0810 22:29:05.525411 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 22:29:05.526495 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:05.513970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43679 (local address 127.1.221.131:33303)
0810 22:29:05.514421 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:05.514465 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:05.516004 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:05.516572 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:05.516581 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:05.516959 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:05.517434 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:05.517447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.518253 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.518255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:05.519118 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:05.519127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.519571 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.519574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:05.519722 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:05.520253 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:05.520270 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:05.521811 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:05.523590 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:05.523596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:05.523611 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:05.523885 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:05.524220 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:05.524224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:05.524226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:05.524375 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:05.524787 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:05.524795 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:05.525046 (+ 251us) client_negotiation.cc:769] Sending connection context
0810 22:29:05.525240 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 22:29:05.525570 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":118,"threads_started":1}
I20260810 22:29:05.527678 8723 heartbeater.cc:347] Connected to a master server at 127.1.221.190:43679
I20260810 22:29:05.527917 8723 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:05.528440 8723 heartbeater.cc:511] Master 127.1.221.190:43679 requested a full tablet report, sending...
I20260810 22:29:05.529500 8213 ts_manager.cc:194] Registered new tserver with Master: a892cd1837b940c99eb4e804dc373034 (127.1.221.131:36655)
I20260810 22:29:05.530105 8213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:33303
I20260810 22:29:05.539904 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/test-admin.keytab.
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:05.558940 1910 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 22:29:05.561952 8730 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:05.560374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.1.221.129:35001)
0810 22:29:05.560737 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:05.560756 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:05.560784 (+ 28us) server_negotiation.cc:407] Connection header received
0810 22:29:05.560851 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:05.560856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:05.561045 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:05.561208 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:05.561729 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44216: BlockingRecv error: recv got EOF from 127.0.0.1:44216 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":130,"threads_started":1}
W20260810 22:29:05.562191 1910 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/krb5cc)
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:05.563212 1910 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:05.564172 1910 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260810 22:29:05.573395 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:05.565063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.1.221.129:35001)
0810 22:29:05.565211 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:05.565215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:05.565301 (+ 86us) server_negotiation.cc:407] Connection header received
0810 22:29:05.565396 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:05.565398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:05.565436 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:05.565500 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:05.566062 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.566788 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.567585 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.567947 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.568029 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:05.570292 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:05.570303 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:05.570316 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:05.570340 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:05.571787 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:05.572196 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:05.572201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:05.572202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:05.572281 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:05.572511 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:05.572514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:05.572515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:05.572784 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:05.572946 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:05.573140 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 22:29:05.573268 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 22:29:05.577348 8730 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:05.576305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44240 (local address 127.1.221.129:35001)
0810 22:29:05.576426 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:05.576430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:05.576585 (+ 155us) server_negotiation.cc:407] Connection header received
0810 22:29:05.576709 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:05.576712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:05.576753 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:05.576842 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:05.577240 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44240: BlockingRecv error: recv got EOF from 127.0.0.1:44240 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 22:29:05.577533 1910 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/krb5cc' of type FILE
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:05.578663 1910 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:05.579664 1910 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260810 22:29:05.587930 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:05.580481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44242 (local address 127.1.221.129:35001)
0810 22:29:05.580633 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:05.580637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:05.580700 (+ 63us) server_negotiation.cc:407] Connection header received
0810 22:29:05.580795 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:05.580797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:05.580831 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:05.580895 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:05.581332 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.581881 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.582731 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.582954 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.583030 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:05.585094 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:05.585097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:05.585099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:05.585121 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:05.586442 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:05.586840 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:05.586843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:05.586844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:05.586917 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:05.587121 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:05.587123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:05.587124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:05.587259 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:05.587440 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:05.587603 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 22:29:05.587696 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 22:29:05.590528 8730 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:05.589658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.1.221.129:35001)
0810 22:29:05.589804 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:05.589813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:05.589861 (+ 48us) server_negotiation.cc:407] Connection header received
0810 22:29:05.589953 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:05.589955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:05.589987 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:05.590048 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:05.590420 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44246: BlockingRecv error: recv got EOF from 127.0.0.1:44246 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 22:29:05.590600 1910 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786400913154531-1910-0/krb5kdc/krb5cc' of type FILE
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:05.591544 1910 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:05.592411 1910 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260810 22:29:05.600183 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:05.593193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44250 (local address 127.1.221.129:35001)
0810 22:29:05.593316 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:05.593320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:05.593392 (+ 72us) server_negotiation.cc:407] Connection header received
0810 22:29:05.593479 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:05.593481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:05.593513 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:05.593626 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:05.594066 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.594530 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.595252 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:05.595452 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:05.595524 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:05.597534 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:05.597537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:05.597539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:05.597560 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:05.598816 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:05.599181 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:05.599183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:05.599185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:05.599254 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:05.599487 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:05.599489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:05.599490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:05.599634 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:05.599745 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:05.599892 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 22:29:05.599973 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 22:29:05.601225 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 8282
I20260810 22:29:05.610255 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 8426
I20260810 22:29:05.618058 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 8578
I20260810 22:29:05.625969 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 8179
2026-08-10T22:29:05Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2279 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8743](info): set up 2 sockets
Aug 10 22:29:05 dist-test-slave-njxd krb5kdc[8743](info): commencing operation
krb5kdc: starting...
W20260810 22:29:06.719760 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:29:06 dist-test-slave-njxd krb5kdc[8743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:29:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.190.keytab.
Entry for principal oryx/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.190.keytab.
I20260810 22:29:06.850428 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40671
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.190.keytab
--principal=oryx/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:06.969159 8757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:06.969458 8757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:06.969502 8757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:06.973734 8757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:06.973814 8757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:06.973834 8757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:06.973850 8757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:06.973865 8757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:29:06.979738 8757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.190.keytab
--principal=oryx/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:06.980960 8757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:06.982122 8757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:06.991362 8762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:06.991390 8765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:06.991371 8763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:06.991371 8757 server_base.cc:1034] running on GCE node
I20260810 22:29:06.992233 8757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:06.993407 8757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:06.994613 8757 hybrid_clock.cc:648] HybridClock initialized: now 1786400946994592 us; error 41 us; skew 500 ppm
Aug 10 22:29:06 dist-test-slave-njxd krb5kdc[8743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400946, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:06.997828 8757 init.cc:377] Logged in from keytab as oryx/127.1.221.190@KRBTEST.COM (short username oryx)
I20260810 22:29:06.998700 8757 webserver.cc:460] Webserver started at http://127.1.221.190:34845/ using document root <none> and password file <none>
I20260810 22:29:06.999239 8757 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:06.999307 8757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:06.999483 8757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:07.001565 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "cde8e631773940d9a966d0acdfad3aec"
format_stamp: "Formatted at 2026-08-10 22:29:06 on dist-test-slave-njxd"
server_key: "3edbf85de904ed5298f4fbe4ee69b22e"
server_key_iv: "d76aeac2ec795b23e8138735017ca3b1"
server_key_version: "encryptionkey@0"
I20260810 22:29:07.002203 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "cde8e631773940d9a966d0acdfad3aec"
format_stamp: "Formatted at 2026-08-10 22:29:06 on dist-test-slave-njxd"
server_key: "3edbf85de904ed5298f4fbe4ee69b22e"
server_key_iv: "d76aeac2ec795b23e8138735017ca3b1"
server_key_version: "encryptionkey@0"
I20260810 22:29:07.006054 8757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:29:07.008570 8772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:07.009929 8757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:29:07.010036 8757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "cde8e631773940d9a966d0acdfad3aec"
format_stamp: "Formatted at 2026-08-10 22:29:06 on dist-test-slave-njxd"
server_key: "3edbf85de904ed5298f4fbe4ee69b22e"
server_key_iv: "d76aeac2ec795b23e8138735017ca3b1"
server_key_version: "encryptionkey@0"
I20260810 22:29:07.010128 8757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:07.035756 8757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:07.038728 8757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:07.038892 8757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:07.046056 8757 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:40671
I20260810 22:29:07.046357 8830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:40671 every 8 connection(s)
I20260810 22:29:07.047088 8757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:29:07.048403 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 8757
I20260810 22:29:07.048543 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:29:07.048893 1910 external_mini_cluster.cc:1428] Setting key 14f1d277c32ec778b2ded1cec4439804
I20260810 22:29:07.050618 8831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:07.057173 8831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec: Bootstrap starting.
Aug 10 22:29:07 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260810 22:29:07.059947 8831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:07.060879 8831 log.cc:824] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:07.063301 8831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec: No bootstrap required, opened a new log
I20260810 22:29:07.063876 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:07.050305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.1.221.190:40671)
0810 22:29:07.050715 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:07.050738 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:07.050784 (+ 46us) server_negotiation.cc:407] Connection header received
0810 22:29:07.051451 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:07.051468 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:07.051791 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:07.052066 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:07.052963 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.054029 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.055192 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.055644 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.055756 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:07.058619 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:07.058627 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:07.058643 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:07.058674 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:07.060694 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:07.061177 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:07.061183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:07.061190 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:07.061340 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:07.061683 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:07.061685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:07.061687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:07.062188 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:07.062382 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:07.062631 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 22:29:07.062926 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":147,"threads_started":1}
I20260810 22:29:07.067358 8831 raft_consensus.cc:348] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } }
I20260810 22:29:07.067587 8831 raft_consensus.cc:374] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:07.067641 8831 raft_consensus.cc:729] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde8e631773940d9a966d0acdfad3aec, State: Initialized, Role: FOLLOWER
I20260810 22:29:07.067981 8831 consensus_queue.cc:260] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } }
I20260810 22:29:07.068096 8831 raft_consensus.cc:388] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:29:07.068141 8831 raft_consensus.cc:482] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:29:07.068208 8831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:07.069060 8831 raft_consensus.cc:504] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } }
I20260810 22:29:07.069464 8831 leader_election.cc:302] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cde8e631773940d9a966d0acdfad3aec; no voters:
I20260810 22:29:07.069859 8831 leader_election.cc:288] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:29:07.070020 8836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:07.070292 8836 raft_consensus.cc:686] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [term 1 LEADER]: Becoming Leader. State: Replica: cde8e631773940d9a966d0acdfad3aec, State: Running, Role: LEADER
I20260810 22:29:07.070698 8836 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } }
I20260810 22:29:07.071303 8831 sys_catalog.cc:564] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:07.072552 8836 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde8e631773940d9a966d0acdfad3aec. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } }
I20260810 22:29:07.072762 8836 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:07.073257 8846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:07.073297 8838 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } }
I20260810 22:29:07.073500 8838 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:07.078052 8846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:07.083757 8846 catalog_manager.cc:1347] Generated new cluster ID: a1199cc48bca4f2c86716b1fe896bbf4
I20260810 22:29:07.083842 8846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:07.102625 8846 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:29:07.103909 8846 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:07.112821 8846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec: Generated new TSK 0
I20260810 22:29:07.113785 8846 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab.
Entry for principal oryx/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab.
I20260810 22:29:07.149138 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40671
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:07.265625 8861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:07.265983 8861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:07.266079 8861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:07.270330 8861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:07.270428 8861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:07.270519 8861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:29:07.276360 8861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:07.277511 8861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:07.278636 8861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:07.288581 8866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:07.288589 8869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:07.288692 8861 server_base.cc:1034] running on GCE node
W20260810 22:29:07.288581 8867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:07.289455 8861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:07.290136 8861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:07.291349 8861 hybrid_clock.cc:648] HybridClock initialized: now 1786400947291325 us; error 42 us; skew 500 ppm
Aug 10 22:29:07 dist-test-slave-njxd krb5kdc[8743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400947, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:07.297326 8861 init.cc:377] Logged in from keytab as oryx/127.1.221.129@KRBTEST.COM (short username oryx)
I20260810 22:29:07.298312 8861 webserver.cc:460] Webserver started at http://127.1.221.129:33781/ using document root <none> and password file <none>
I20260810 22:29:07.298905 8861 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:07.299019 8861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:07.299269 8861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:07.301913 8861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "a4013ce0110a486eb0e2e01623774c2c"
format_stamp: "Formatted at 2026-08-10 22:29:07 on dist-test-slave-njxd"
server_key: "9aebbd169c8ecb1bf4d1897a755aaaa7"
server_key_iv: "e431fd319943de59ddd416b178fce116"
server_key_version: "encryptionkey@0"
I20260810 22:29:07.302515 8861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "a4013ce0110a486eb0e2e01623774c2c"
format_stamp: "Formatted at 2026-08-10 22:29:07 on dist-test-slave-njxd"
server_key: "9aebbd169c8ecb1bf4d1897a755aaaa7"
server_key_iv: "e431fd319943de59ddd416b178fce116"
server_key_version: "encryptionkey@0"
I20260810 22:29:07.306371 8861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 22:29:07.308892 8876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:07.310215 8861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:29:07.310360 8861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "a4013ce0110a486eb0e2e01623774c2c"
format_stamp: "Formatted at 2026-08-10 22:29:07 on dist-test-slave-njxd"
server_key: "9aebbd169c8ecb1bf4d1897a755aaaa7"
server_key_iv: "e431fd319943de59ddd416b178fce116"
server_key_version: "encryptionkey@0"
I20260810 22:29:07.310474 8861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:07.349853 8861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:07.353011 8861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:07.353246 8861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:07.353945 8861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:07.354858 8861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:07.354938 8861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:07.355006 8861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:07.355058 8861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:07.364485 8861 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:33893
I20260810 22:29:07.364511 9003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:33893 every 8 connection(s)
I20260810 22:29:07.365463 8861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:29:07.367432 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 8861
I20260810 22:29:07.367570 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:29:07.367895 1910 external_mini_cluster.cc:1428] Setting key b0c1973cb6a4e131defba3505f70808d
Aug 10 22:29:07 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400947, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260810 22:29:07.381098 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:07.367480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:51495 (local address 127.1.221.190:40671)
0810 22:29:07.367654 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:07.367657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:07.368833 (+ 1176us) server_negotiation.cc:407] Connection header received
0810 22:29:07.370509 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:07.370513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:07.370568 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:07.370647 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:07.372115 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.372785 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.374185 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.374437 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.374569 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:07.377271 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:07.377276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:07.377282 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:07.377316 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:07.379095 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:07.379608 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:07.379611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:07.379612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:07.379688 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:07.380048 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:07.380050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:07.380051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:07.380230 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:07.380323 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:07.380860 (+ 537us) server_negotiation.cc:299] Negotiation successful
0810 22:29:07.380970 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.130@KRBTEST.COM" created.
I20260810 22:29:07.382131 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:07.368023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.129:51495)
0810 22:29:07.368664 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:07.368711 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:07.370282 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:07.370834 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:07.370841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:07.371219 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:07.371960 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:07.371979 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.372918 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.372921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:07.374048 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:07.374059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.374615 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.374619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:07.374826 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:07.375500 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:07.375517 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:07.377073 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:07.379215 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:07.379220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:07.379234 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:07.379510 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:07.379775 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:07.379778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:07.379779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:07.379922 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:07.380352 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:07.380358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:07.380622 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 22:29:07.380802 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 22:29:07.381076 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":167,"threads_started":1}
I20260810 22:29:07.383592 9004 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40671
I20260810 22:29:07.383891 9004 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:07.384423 9004 heartbeater.cc:511] Master 127.1.221.190:40671 requested a full tablet report, sending...
I20260810 22:29:07.386127 8792 ts_manager.cc:194] Registered new tserver with Master: a4013ce0110a486eb0e2e01623774c2c (127.1.221.129:33893)
I20260810 22:29:07.387902 8792 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:51495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab.
Entry for principal oryx/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab.
I20260810 22:29:07.400597 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40671
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:07.518216 9009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:07.518509 9009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:07.518554 9009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:07.522614 9009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:07.522684 9009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:07.522744 9009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:29:07.528136 9009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:07.529253 9009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:07.530428 9009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:07.540508 9015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:07.540518 9014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:07.540524 9017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:07.540859 9009 server_base.cc:1034] running on GCE node
I20260810 22:29:07.541457 9009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:07.542166 9009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:07.543334 9009 hybrid_clock.cc:648] HybridClock initialized: now 1786400947543313 us; error 35 us; skew 500 ppm
Aug 10 22:29:07 dist-test-slave-njxd krb5kdc[8743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400947, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:07.546283 9009 init.cc:377] Logged in from keytab as oryx/127.1.221.130@KRBTEST.COM (short username oryx)
I20260810 22:29:07.547284 9009 webserver.cc:460] Webserver started at http://127.1.221.130:39553/ using document root <none> and password file <none>
I20260810 22:29:07.547899 9009 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:07.547991 9009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:07.548233 9009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:07.550534 9009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "9e00f7e5f7ca459a866c8b78fc0df48a"
format_stamp: "Formatted at 2026-08-10 22:29:07 on dist-test-slave-njxd"
server_key: "6cc9d313407c7427e1faa5ccd62544b9"
server_key_iv: "b308281f9d80d1f9dc8fc93f2bd958b8"
server_key_version: "encryptionkey@0"
I20260810 22:29:07.551195 9009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "9e00f7e5f7ca459a866c8b78fc0df48a"
format_stamp: "Formatted at 2026-08-10 22:29:07 on dist-test-slave-njxd"
server_key: "6cc9d313407c7427e1faa5ccd62544b9"
server_key_iv: "b308281f9d80d1f9dc8fc93f2bd958b8"
server_key_version: "encryptionkey@0"
I20260810 22:29:07.555428 9009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 22:29:07.558260 9026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:07.559872 9009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:29:07.560014 9009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "9e00f7e5f7ca459a866c8b78fc0df48a"
format_stamp: "Formatted at 2026-08-10 22:29:07 on dist-test-slave-njxd"
server_key: "6cc9d313407c7427e1faa5ccd62544b9"
server_key_iv: "b308281f9d80d1f9dc8fc93f2bd958b8"
server_key_version: "encryptionkey@0"
I20260810 22:29:07.560148 9009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:07.589123 9009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:07.592248 9009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:07.592473 9009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:07.593143 9009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:07.594082 9009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:07.594132 9009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:07.594205 9009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:07.594246 9009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:07.604168 9009 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:39725
I20260810 22:29:07.604192 9146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:39725 every 8 connection(s)
I20260810 22:29:07.605123 9009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:29:07.608493 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 9009
I20260810 22:29:07.608624 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:29:07.609007 1910 external_mini_cluster.cc:1428] Setting key 46e3f9396a565e0dcbd08fe6fc0f6e93
Aug 10 22:29:07 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400947, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260810 22:29:07.621006 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:07.607066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:42239 (local address 127.1.221.190:40671)
0810 22:29:07.607216 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:07.607219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:07.608064 (+ 845us) server_negotiation.cc:407] Connection header received
0810 22:29:07.609898 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:07.609902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:07.609950 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:07.610041 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:07.611705 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.612341 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.614059 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.614278 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.614425 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:07.617286 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:07.617295 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:07.617297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:07.617322 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:07.618841 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:07.619387 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:07.619396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:07.619397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:07.619508 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:07.619864 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:07.619869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:07.619870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:07.620060 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:07.620173 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:07.620697 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 22:29:07.620836 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:29:07.621941 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:07.607409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.130:42239)
0810 22:29:07.607889 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:07.607935 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:07.609655 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:07.610212 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:07.610219 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:07.610788 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:07.611536 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:07.611562 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.612506 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.612512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:07.613811 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:07.613822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.614473 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.614477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:07.614734 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:07.615410 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:07.615429 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:07.617109 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:07.618972 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:07.618977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:07.618994 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:07.619261 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:07.619602 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:07.619605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:07.619608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:07.619760 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:07.620173 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:07.620182 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:07.620451 (+ 269us) client_negotiation.cc:769] Sending connection context
0810 22:29:07.620653 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 22:29:07.621003 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.131@KRBTEST.COM" created.
I20260810 22:29:07.623221 9148 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40671
I20260810 22:29:07.623497 9148 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:07.624131 9148 heartbeater.cc:511] Master 127.1.221.190:40671 requested a full tablet report, sending...
I20260810 22:29:07.625341 8792 ts_manager.cc:194] Registered new tserver with Master: 9e00f7e5f7ca459a866c8b78fc0df48a (127.1.221.130:39725)
I20260810 22:29:07.626024 8792 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:42239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab.
Entry for principal oryx/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab.
I20260810 22:29:07.640333 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40671
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:07.756732 9153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:07.757050 9153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:07.757133 9153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:07.761230 9153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:07.761334 9153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:07.761432 9153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:29:07.766968 9153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40671
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:07.768119 9153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:07.769203 9153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:07.779139 9162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:07.779139 9159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:07.779261 9153 server_base.cc:1034] running on GCE node
W20260810 22:29:07.779147 9160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:07.779944 9153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:07.780596 9153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:07.781785 9153 hybrid_clock.cc:648] HybridClock initialized: now 1786400947781750 us; error 46 us; skew 500 ppm
Aug 10 22:29:07 dist-test-slave-njxd krb5kdc[8743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400947, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:07.784627 9153 init.cc:377] Logged in from keytab as oryx/127.1.221.131@KRBTEST.COM (short username oryx)
I20260810 22:29:07.785481 9153 webserver.cc:460] Webserver started at http://127.1.221.131:34679/ using document root <none> and password file <none>
I20260810 22:29:07.786091 9153 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:07.786188 9153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:07.786423 9153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:07.788674 9153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "3aa8d37a0d844500aa9fe801dd62efff"
format_stamp: "Formatted at 2026-08-10 22:29:07 on dist-test-slave-njxd"
server_key: "e81825707d40a2a5223fe30d0c27a187"
server_key_iv: "f44d53059a7c1f0da143a8f91999579b"
server_key_version: "encryptionkey@0"
I20260810 22:29:07.789268 9153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "3aa8d37a0d844500aa9fe801dd62efff"
format_stamp: "Formatted at 2026-08-10 22:29:07 on dist-test-slave-njxd"
server_key: "e81825707d40a2a5223fe30d0c27a187"
server_key_iv: "f44d53059a7c1f0da143a8f91999579b"
server_key_version: "encryptionkey@0"
I20260810 22:29:07.793105 9153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:29:07.795584 9173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:07.796810 9153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:29:07.796944 9153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "3aa8d37a0d844500aa9fe801dd62efff"
format_stamp: "Formatted at 2026-08-10 22:29:07 on dist-test-slave-njxd"
server_key: "e81825707d40a2a5223fe30d0c27a187"
server_key_iv: "f44d53059a7c1f0da143a8f91999579b"
server_key_version: "encryptionkey@0"
I20260810 22:29:07.797055 9153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:07.814926 9153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:07.818234 9153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:07.818463 9153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:07.819049 9153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:07.819856 9153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:07.819928 9153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:07.819994 9153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:07.820041 9153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:07.828879 9153 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:43113
I20260810 22:29:07.828903 9294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:43113 every 8 connection(s)
I20260810 22:29:07.829783 9153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:29:07.838390 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 9153
I20260810 22:29:07.838492 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:29:07.838794 1910 external_mini_cluster.cc:1428] Setting key c2320f5a576a888f0815c927260d8bad
Aug 10 22:29:07 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400947, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260810 22:29:07.843466 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:07.831660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:36085 (local address 127.1.221.190:40671)
0810 22:29:07.831813 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:07.831817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:07.832669 (+ 852us) server_negotiation.cc:407] Connection header received
0810 22:29:07.834024 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:07.834027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:07.834076 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:07.834152 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:07.835318 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.835845 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.836945 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.837152 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.837226 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:07.839974 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:07.839978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:07.839980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:07.840009 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:07.841644 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:07.842130 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:07.842133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:07.842134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:07.842207 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:07.842525 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:07.842527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:07.842528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:07.842703 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:07.842780 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:07.843229 (+ 449us) server_negotiation.cc:299] Negotiation successful
0810 22:29:07.843338 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:29:07.844331 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:07.832034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.131:36085)
0810 22:29:07.832498 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:07.832542 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:07.833813 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:07.834323 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:07.834330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:07.834704 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:07.835188 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:07.835201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.835948 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.835951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:07.836839 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:07.836847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:07.837284 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:07.837286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:07.837430 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:07.838082 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:07.838098 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:07.839795 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:07.841765 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:07.841776 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:07.841788 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:07.842045 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:07.842309 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:07.842311 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:07.842312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:07.842444 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:07.842798 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:07.842802 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:07.843028 (+ 226us) client_negotiation.cc:769] Sending connection context
0810 22:29:07.843181 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 22:29:07.843433 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":123,"threads_started":1}
I20260810 22:29:07.845486 9295 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40671
I20260810 22:29:07.845746 9295 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:07.846243 9295 heartbeater.cc:511] Master 127.1.221.190:40671 requested a full tablet report, sending...
I20260810 22:29:07.847252 8792 ts_manager.cc:194] Registered new tserver with Master: 3aa8d37a0d844500aa9fe801dd62efff (127.1.221.131:43113)
I20260810 22:29:07.847839 8792 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:36085
I20260810 22:29:07.852761 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.221.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.189.keytab.
Entry for principal oryx/127.1.221.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.189.keytab.
I20260810 22:29:07.881333 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:45551
--webserver_interface=127.1.221.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.189.keytab
--principal=oryx/127.1.221.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40671,127.1.221.189:45551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:07.995400 9301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:07.995730 9301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:07.995818 9301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:07.999873 9301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:07.999954 9301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:08.000006 9301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:08.000047 9301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:08.000082 9301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:29:08.005493 9301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40671,127.1.221.189:45551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.189.keytab
--principal=oryx/127.1.221.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:45551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.221.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
W20260810 22:29:08.005851 9301 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.221.190:40671,127.1.221.189:45551'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 22:29:08.006812 9301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:08.007905 9301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:08.016708 9307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:08.016607 9309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:08.016616 9306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:08.017293 9301 server_base.cc:1034] running on GCE node
I20260810 22:29:08.017982 9301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:08.019104 9301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:08.020268 9301 hybrid_clock.cc:648] HybridClock initialized: now 1786400948020249 us; error 28 us; skew 500 ppm
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:08.023008 9301 init.cc:377] Logged in from keytab as oryx/127.1.221.189@KRBTEST.COM (short username oryx)
I20260810 22:29:08.023833 9301 webserver.cc:460] Webserver started at http://127.1.221.189:38151/ using document root <none> and password file <none>
I20260810 22:29:08.024365 9301 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:08.024422 9301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:08.024646 9301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:08.026741 9301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/data/instance:
uuid: "1c7324d97a184a1ba4e86eb31802017e"
format_stamp: "Formatted at 2026-08-10 22:29:08 on dist-test-slave-njxd"
server_key: "eab84a1c4975eeb09c5f7952eb6f77f5"
server_key_iv: "7807aeaf378c256def6de02d9f8772f9"
server_key_version: "encryptionkey@0"
I20260810 22:29:08.027281 9301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/wal/instance:
uuid: "1c7324d97a184a1ba4e86eb31802017e"
format_stamp: "Formatted at 2026-08-10 22:29:08 on dist-test-slave-njxd"
server_key: "eab84a1c4975eeb09c5f7952eb6f77f5"
server_key_iv: "7807aeaf378c256def6de02d9f8772f9"
server_key_version: "encryptionkey@0"
I20260810 22:29:08.031066 9301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:29:08.033456 9318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:08.034797 9301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:29:08.034945 9301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/wal
uuid: "1c7324d97a184a1ba4e86eb31802017e"
format_stamp: "Formatted at 2026-08-10 22:29:08 on dist-test-slave-njxd"
server_key: "eab84a1c4975eeb09c5f7952eb6f77f5"
server_key_iv: "7807aeaf378c256def6de02d9f8772f9"
server_key_version: "encryptionkey@0"
I20260810 22:29:08.035065 9301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:08.069069 9301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:08.072186 9301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:08.072463 9301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:08.080071 9301 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.189:45551
I20260810 22:29:08.080374 9377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.189:45551 every 8 connection(s)
I20260810 22:29:08.081135 9301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/data/info.pb
I20260810 22:29:08.084005 9379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:08.087929 9379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:29:08.089092 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 9301
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.189@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
I20260810 22:29:08.102232 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.090192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.1.221.190:40671)
0810 22:29:08.090357 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.090362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.091195 (+ 833us) server_negotiation.cc:407] Connection header received
0810 22:29:08.092156 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.092160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.092228 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.092348 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.093371 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.093934 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.095076 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.095308 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.095388 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.098094 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.098100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.098106 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.098138 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.099879 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.100579 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.100582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.100583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.100656 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.101010 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.101015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.101022 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.101287 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.101401 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.101943 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.102058 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 22:29:08.103025 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.090542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:35544)
0810 22:29:08.091014 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.091082 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.091937 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.092418 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.092427 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.092863 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.093222 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.093238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.094055 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.094059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.094949 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.094958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.095449 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.095452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.095681 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.096300 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.096315 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.097944 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.100049 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.100055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.100065 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.100481 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.100761 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.100765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.100767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.100909 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.101400 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.101407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.101707 (+ 300us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.101898 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.102201 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":64,"spinlock_wait_cycles":5376,"thread_start_us":129,"threads_started":1}
I20260810 22:29:08.103027 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.091300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.1.221.189:45551)
0810 22:29:08.091583 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.091596 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.091630 (+ 34us) server_negotiation.cc:407] Connection header received
0810 22:29:08.091951 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.091972 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.092165 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.092371 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.093095 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.094253 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.095106 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.095577 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.095686 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.098655 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.098667 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.098678 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.098699 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.100636 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.101085 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.101093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.101096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.101185 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.101466 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.101470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.101472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.101775 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.101991 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.102175 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.102597 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":167,"thread_start_us":100,"threads_started":1}
I20260810 22:29:08.104359 9379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.189@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
I20260810 22:29:08.112455 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.104846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:45551 (local address 127.0.0.1:46508)
0810 22:29:08.105212 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.105226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.105293 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.105454 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.105456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.105631 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.105727 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.105732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.106482 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.106484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.107004 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.107010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.107981 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.107984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.108060 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.108448 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.108459 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.109750 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.111187 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.111190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.111191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.111388 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.111621 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.111623 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.111625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.111697 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.112012 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.112014 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.112087 (+ 73us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.112165 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.112286 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310}
I20260810 22:29:08.112463 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.104936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.1.221.189:45551)
0810 22:29:08.105037 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.105040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.105290 (+ 250us) server_negotiation.cc:407] Connection header received
0810 22:29:08.105349 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.105351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.105384 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.105442 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.105822 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.106377 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.107094 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.107876 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.107963 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.109876 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.109879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.109881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.109902 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.111090 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.111473 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.111476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.111477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.111545 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.111773 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.111775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.111776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.111933 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.112008 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.112196 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.112342 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":40}
I20260810 22:29:08.115769 9379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: Bootstrap starting.
I20260810 22:29:08.118309 9379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:08.119050 9379 log.cc:824] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:08.121032 9379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: No bootstrap required, opened a new log
I20260810 22:29:08.124163 9379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } }
I20260810 22:29:08.124366 9379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:08.124449 9379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7324d97a184a1ba4e86eb31802017e, State: Initialized, Role: FOLLOWER
I20260810 22:29:08.124868 9379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } }
I20260810 22:29:08.125365 9385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } } }
I20260810 22:29:08.125504 9385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:29:08.126117 9379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: configured and running, proceeding with master startup.
W20260810 22:29:08.130843 9397 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:29:08.130939 9397 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:29:08.133612 9301 master_runner.cc:428] Detected that this master 1c7324d97a184a1ba4e86eb31802017e is joining an existing cluster
I20260810 22:29:08.133678 9301 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.221.189:45551
I20260810 22:29:08.144984 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.135683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.1.221.190:40671)
0810 22:29:08.135835 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.135840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.136453 (+ 613us) server_negotiation.cc:407] Connection header received
0810 22:29:08.136608 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.136612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.136672 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.136777 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.137535 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.138309 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.139320 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.139667 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.139804 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.141097 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.141101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.141103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.141128 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.142976 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.143576 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.143583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.143587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.143682 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.143978 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.143983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.143986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.144247 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.144361 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.144685 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.144822 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:29:08.144981 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.136133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.1.221.189:45551)
0810 22:29:08.136252 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.136257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.136501 (+ 244us) server_negotiation.cc:407] Connection header received
0810 22:29:08.136676 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.136679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.136735 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.136814 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.137413 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.138191 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.139445 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.139792 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.139863 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.141250 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.141255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.141262 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.141293 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.143095 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.143603 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.143606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.143608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.143680 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.144192 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.144195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.144197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.144404 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.144504 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.144768 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.144865 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:29:08.144995 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.135618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:35552)
0810 22:29:08.136350 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.136366 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.136472 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.136786 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.136789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.137067 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.137190 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.137196 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.138409 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.138412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.139190 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.139199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.139808 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.139811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.139938 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.140491 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.140512 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.140937 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.143104 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.143108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.143110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.143461 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.143786 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.143790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.143791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.143880 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.144346 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.144349 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.144480 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.144599 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.144762 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":19,"thread_start_us":96,"threads_started":1}
I20260810 22:29:08.145455 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.136065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:45551 (local address 127.0.0.1:46518)
0810 22:29:08.136408 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.136440 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.136551 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.136931 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.136934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.137191 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.137301 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.137309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.138333 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.138337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.139298 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.139308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.140024 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.140030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.140144 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.140626 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.140640 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.141120 (+ 480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.143179 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.143181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.143183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.143475 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.143996 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.143998 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.144000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.144089 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.144504 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.144507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.144599 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.145191 (+ 592us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.145336 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":21,"thread_start_us":113,"threads_started":1}
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400946, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.190@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
I20260810 22:29:08.160193 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.150372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.1.221.189:45551)
0810 22:29:08.150514 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.150519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.150910 (+ 391us) server_negotiation.cc:407] Connection header received
0810 22:29:08.151089 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.151091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.151134 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.151199 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.151965 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.152428 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.153687 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.153895 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.154021 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.156552 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.156556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.156559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.156585 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.158289 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.158897 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.158902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.158904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.158986 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.159271 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.159275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.159278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.159455 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.159564 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.159905 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.160028 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:29:08.160214 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.150341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:45551 (local address 127.0.0.1:46528)
0810 22:29:08.150763 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.150822 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.150958 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.151261 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.151267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.151707 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.151855 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.151863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.152540 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.152543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.153555 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.153566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.154028 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.154031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.154155 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.154669 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.154687 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.156343 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.158410 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.158424 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.158434 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.158788 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.159083 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.159087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.159089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.159175 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.159566 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.159577 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.159698 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.159851 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.160051 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":123,"threads_started":1}
I20260810 22:29:08.161875 8792 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.221.189:45551: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e"
member_type: NON_VOTER
last_known_addr {
host: "127.1.221.189"
port: 45551
}
attrs {
promote: true
}
}
dest_uuid: "cde8e631773940d9a966d0acdfad3aec"
cas_config_opid_index: -1
I20260810 22:29:08.162644 8792 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true } }
I20260810 22:29:08.165154 9410 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1c7324d97a184a1ba4e86eb31802017e (127.1.221.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true } } }
I20260810 22:29:08.165978 9348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:08.166931 9410 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.221.189:45551
WARNING: no policy specified for oryx/127.1.221.188@KRBTEST.COM; defaulting to no policy
I20260810 22:29:08.167397 9301 master.cc:537] Master@127.1.221.189:45551 shutting down...
I20260810 22:29:08.167596 9348 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 FOLLOWER]: Refusing update from remote peer cde8e631773940d9a966d0acdfad3aec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 22:29:08.168272 9410 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true } } }
I20260810 22:29:08.168375 9410 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:08.168767 9409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:29:08.171598 8774 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 22:29:08.172562 8774 consensus_peers.cc:487] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec -> Peer 1c7324d97a184a1ba4e86eb31802017e (127.1.221.189:45551): Couldn't send request to peer 1c7324d97a184a1ba4e86eb31802017e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:29:08.172737 9301 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:08.173070 9301 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 FOLLOWER]: Raft consensus is shut down!
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.188@KRBTEST.COM" created.
I20260810 22:29:08.173182 9301 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: stopping tablet replica
I20260810 22:29:08.179090 9301 master.cc:552] Master@127.1.221.189:45551 shutdown complete.
I20260810 22:29:08.179152 9301 master_runner.cc:305] Clearing existing system tablet
I20260810 22:29:08.179474 9301 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:08.179540 9301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:08.181182 9301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 22:29:08.182271 9415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:08.182533 9301 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:08.182749 9301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 22:29:08.182819 9301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/wal
uuid: "1c7324d97a184a1ba4e86eb31802017e"
format_stamp: "Formatted at 2026-08-10 22:29:08 on dist-test-slave-njxd"
server_key: "eab84a1c4975eeb09c5f7952eb6f77f5"
server_key_iv: "7807aeaf378c256def6de02d9f8772f9"
server_key_version: "encryptionkey@0"
I20260810 22:29:08.185204 9301 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:08.188334 9301 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 22:29:08.188479 9301 log.cc:1196] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 22:29:08.188778 9301 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: Deleting consensus metadata
I20260810 22:29:08.189006 9301 master_runner.cc:315] Copying system tablet from 127.1.221.190:40671
I20260810 22:29:08.190414 9301 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: tablet copy: Beginning tablet copy session from remote peer at address 127.1.221.190:40671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.188.keytab.
Entry for principal oryx/127.1.221.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.188.keytab.
I20260810 22:29:08.193980 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:35943
--webserver_interface=127.1.221.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.188.keytab
--principal=oryx/127.1.221.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40671,127.1.221.189:45551,127.1.221.188:35943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:29:08.199942 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.191062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:35560)
0810 22:29:08.191361 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.191377 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.191491 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.191727 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.191729 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.191953 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.192069 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.192075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.193191 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.193195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.194125 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.194136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.194870 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.194873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.194968 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.195542 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.195557 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.196008 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.198187 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.198190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.198193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.198439 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.198768 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.198772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.198774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.198866 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.199356 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.199359 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.199458 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.199563 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.199697 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
I20260810 22:29:08.200057 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.191154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.1.221.190:40671)
0810 22:29:08.191294 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.191299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.191470 (+ 171us) server_negotiation.cc:407] Connection header received
0810 22:29:08.191566 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.191569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.191622 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.191727 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.192191 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.193040 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.194270 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.194626 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.194698 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.196183 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.196189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.196194 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.196228 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.198069 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.198548 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.198554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.198559 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.198654 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.198969 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.198975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.198980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.199235 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.199361 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.199657 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.199820 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:29:08.200667 8814 tablet_copy_service.cc:140] P cde8e631773940d9a966d0acdfad3aec: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1c7324d97a184a1ba4e86eb31802017e ({username='oryx', principal='oryx/127.1.221.189@KRBTEST.COM'} at 127.0.0.1:35560)
I20260810 22:29:08.200870 8814 tablet_copy_service.cc:161] P cde8e631773940d9a966d0acdfad3aec: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1c7324d97a184a1ba4e86eb31802017e at {username='oryx', principal='oryx/127.1.221.189@KRBTEST.COM'} at 127.0.0.1:35560: session id = 1c7324d97a184a1ba4e86eb31802017e-00000000000000000000000000000000
I20260810 22:29:08.202860 8814 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:29:08.205904 9301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 22:29:08.208694 9301 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: tablet copy: Starting download of 0 data blocks...
I20260810 22:29:08.208896 9301 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: tablet copy: Starting download of 1 WAL segments...
I20260810 22:29:08.210758 9301 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:29:08.212560 8814 tablet_copy_service.cc:342] P cde8e631773940d9a966d0acdfad3aec: Request end of tablet copy session 1c7324d97a184a1ba4e86eb31802017e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.221.189@KRBTEST.COM'} at 127.0.0.1:35560
I20260810 22:29:08.212677 8814 tablet_copy_service.cc:433] P cde8e631773940d9a966d0acdfad3aec: ending tablet copy session 1c7324d97a184a1ba4e86eb31802017e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1c7324d97a184a1ba4e86eb31802017e
I20260810 22:29:08.220788 9301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:08.222554 9425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:08.226452 9426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:08.226866 9301 server_base.cc:1034] running on GCE node
W20260810 22:29:08.226997 9428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:08.227237 9301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:08.227586 9301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:08.228735 9301 hybrid_clock.cc:648] HybridClock initialized: now 1786400948228715 us; error 47 us; skew 500 ppm
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:08.230482 9301 init.cc:377] Logged in from keytab as oryx/127.1.221.189@KRBTEST.COM (short username oryx)
I20260810 22:29:08.230985 9301 webserver.cc:460] Webserver started at http://127.1.221.189:40605/ using document root <none> and password file <none>
I20260810 22:29:08.231194 9301 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:08.231237 9301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:08.232741 9301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 22:29:08.233879 9435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:08.234170 9301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 22:29:08.234251 9301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/wal
uuid: "1c7324d97a184a1ba4e86eb31802017e"
format_stamp: "Formatted at 2026-08-10 22:29:08 on dist-test-slave-njxd"
server_key: "eab84a1c4975eeb09c5f7952eb6f77f5"
server_key_iv: "7807aeaf378c256def6de02d9f8772f9"
server_key_version: "encryptionkey@0"
I20260810 22:29:08.234335 9301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:08.243606 9301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:08.245848 9301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:08.251119 9301 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.189:45551
I20260810 22:29:08.251775 9301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-1/data/info.pb
I20260810 22:29:08.261821 9495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.189:45551 every 8 connection(s)
I20260810 22:29:08.263733 9496 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:29:08.264900 9496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: Bootstrap starting.
I20260810 22:29:08.276583 9496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:29:08.276952 9496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: Bootstrap complete.
I20260810 22:29:08.277631 9496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true } }
I20260810 22:29:08.277819 9496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c7324d97a184a1ba4e86eb31802017e, State: Initialized, Role: LEARNER
I20260810 22:29:08.277949 9496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true } }
I20260810 22:29:08.278370 9500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true } } }
I20260810 22:29:08.278481 9496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:08.278573 9500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: This master's current role is: LEARNER
I20260810 22:29:08.281867 9513 catalog_manager.cc:1259] Loaded cluster ID: a1199cc48bca4f2c86716b1fe896bbf4
I20260810 22:29:08.281994 9513 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: loading cluster ID for follower catalog manager: success
I20260810 22:29:08.284196 9513 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: acquiring CA information for follower catalog manager: success
I20260810 22:29:08.285727 9513 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 22:29:08.330744 9424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:08.331013 9424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:08.331058 9424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:08.335290 9424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:08.335352 9424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:08.335399 9424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:08.335429 9424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:08.335449 9424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:29:08.340691 9424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40671,127.1.221.189:45551,127.1.221.188:35943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.188.keytab
--principal=oryx/127.1.221.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:35943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.221.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:08.341846 9424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:08.342891 9424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:08.351375 9519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:08.351363 9518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:08.351361 9521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:08.351702 9424 server_base.cc:1034] running on GCE node
I20260810 22:29:08.352257 9424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:08.353331 9424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:08.354523 9424 hybrid_clock.cc:648] HybridClock initialized: now 1786400948354506 us; error 36 us; skew 500 ppm
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:08.357347 9424 init.cc:377] Logged in from keytab as oryx/127.1.221.188@KRBTEST.COM (short username oryx)
I20260810 22:29:08.358295 9424 webserver.cc:460] Webserver started at http://127.1.221.188:45503/ using document root <none> and password file <none>
I20260810 22:29:08.358862 9424 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:08.358958 9424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:08.359189 9424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:08.361227 9424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/data/instance:
uuid: "0da636ffd07b4335b5158a89e987d75e"
format_stamp: "Formatted at 2026-08-10 22:29:08 on dist-test-slave-njxd"
server_key: "ccaf7efb4defa1b0826b66c589213356"
server_key_iv: "a7e0d66e3df8dc886b9e85520d79efde"
server_key_version: "encryptionkey@0"
I20260810 22:29:08.361856 9424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/wal/instance:
uuid: "0da636ffd07b4335b5158a89e987d75e"
format_stamp: "Formatted at 2026-08-10 22:29:08 on dist-test-slave-njxd"
server_key: "ccaf7efb4defa1b0826b66c589213356"
server_key_iv: "a7e0d66e3df8dc886b9e85520d79efde"
server_key_version: "encryptionkey@0"
I20260810 22:29:08.365965 9424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 22:29:08.368885 9528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:08.370165 9424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260810 22:29:08.370261 9424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/wal
uuid: "0da636ffd07b4335b5158a89e987d75e"
format_stamp: "Formatted at 2026-08-10 22:29:08 on dist-test-slave-njxd"
server_key: "ccaf7efb4defa1b0826b66c589213356"
server_key_iv: "a7e0d66e3df8dc886b9e85520d79efde"
server_key_version: "encryptionkey@0"
I20260810 22:29:08.370347 9424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:08.390695 9004 heartbeater.cc:503] Master 127.1.221.190:40671 was elected leader, sending a full tablet report...
I20260810 22:29:08.399515 9424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:08.402992 9424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:08.403191 9424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:08.410491 9424 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.188:35943
I20260810 22:29:08.410791 9585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.188:35943 every 8 connection(s)
I20260810 22:29:08.411464 9424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/data/info.pb
I20260810 22:29:08.412519 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 9424
I20260810 22:29:08.415132 9586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:08.419437 9586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.188@KRBTEST.COM
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.188@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260810 22:29:08.427717 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.414160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.1.221.188:35943)
0810 22:29:08.414636 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.414658 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.414705 (+ 47us) server_negotiation.cc:407] Connection header received
0810 22:29:08.415472 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.415490 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.415818 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.416090 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.416994 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.418007 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.419143 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.419604 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.419715 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.422733 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.422740 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.422752 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.422779 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.424743 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.425237 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.425243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.425249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.425399 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.425750 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.425755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.425757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.426156 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.426409 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.426648 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.426921 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":152,"threads_started":1}
I20260810 22:29:08.431613 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.421626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.1.221.190:40671)
0810 22:29:08.421762 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.421768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.421997 (+ 229us) server_negotiation.cc:407] Connection header received
0810 22:29:08.422113 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.422116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.422182 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.422270 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.423001 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.423505 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.424777 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.425149 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.425254 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.427549 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.427553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.427558 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.427588 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.429805 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.430276 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.430282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.430285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.430374 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.430670 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.430674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.430676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.430864 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.430977 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.431325 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.431445 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:29:08.431650 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.421452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:35576)
0810 22:29:08.421859 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.421908 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.422025 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.422321 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.422326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.422697 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.422865 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.422873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.423632 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.423636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.424643 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.424656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.425260 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.425263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.425381 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.425966 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.425978 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.427405 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.429923 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.429932 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.429937 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.430167 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.430481 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.430484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.430486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.430574 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.430988 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.430996 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.431111 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.431276 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.431451 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":159,"threads_started":1}
I20260810 22:29:08.432688 9586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.188@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
I20260810 22:29:08.440590 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.433139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:45551 (local address 127.0.0.1:46530)
0810 22:29:08.433259 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.433271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.433338 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.433868 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.433871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.434029 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.434113 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.434119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.435096 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.435102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.435624 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.435632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.436137 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.436141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.436239 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.436613 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.436624 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.437830 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.439325 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.439328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.439329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.439508 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.439815 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.439817 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.439818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.439889 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.440232 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.440233 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.440305 (+ 72us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.440384 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.440477 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 22:29:08.440820 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.433312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.1.221.189:45551)
0810 22:29:08.433615 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.433621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.433635 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:29:08.433676 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.433678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.433718 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.433842 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.434237 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.434982 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.435758 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.436021 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.436135 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.437956 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.437959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.437961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.437984 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.439222 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.439626 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.439629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.439630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.439719 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.439970 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.439972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.439973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.440150 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.440229 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.440443 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.440548 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260810 22:29:08.441545 9586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.188@KRBTEST.COM for oryx/127.1.221.188@KRBTEST.COM
I20260810 22:29:08.449090 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.442088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35943 (local address 127.0.0.1:58526)
0810 22:29:08.442202 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.442218 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.442291 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.442495 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.442497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.442622 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.442689 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.442693 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.443720 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.443723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.444210 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.444218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.444609 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.444612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.444667 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.445142 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.445154 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.446335 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.447895 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.447898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.447900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.448080 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.448324 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.448326 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.448327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.448399 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.448713 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.448715 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.448792 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.448870 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.448964 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 22:29:08.449169 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.442201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.1.221.188:35943)
0810 22:29:08.442307 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.442311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.442348 (+ 37us) server_negotiation.cc:407] Connection header received
0810 22:29:08.442376 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.442378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.442411 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.442469 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.442781 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.443626 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.444307 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.444517 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.444595 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.446467 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.446471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.446473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.446495 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.447794 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.448171 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.448174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.448175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.448246 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.448473 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.448475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.448476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.448637 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.448709 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.448928 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.449014 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 22:29:08.452073 9586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: Bootstrap starting.
I20260810 22:29:08.455080 9586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:08.456123 9586 log.cc:824] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:08.458125 9586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: No bootstrap required, opened a new log
I20260810 22:29:08.461102 9586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } }
I20260810 22:29:08.461302 9586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:08.461385 9586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da636ffd07b4335b5158a89e987d75e, State: Initialized, Role: FOLLOWER
I20260810 22:29:08.461866 9586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } }
I20260810 22:29:08.462330 9593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } } }
I20260810 22:29:08.462472 9593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:29:08.463075 9586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: configured and running, proceeding with master startup.
W20260810 22:29:08.468180 9604 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:29:08.468286 9604 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:29:08.471405 9424 master_runner.cc:428] Detected that this master 0da636ffd07b4335b5158a89e987d75e is joining an existing cluster
I20260810 22:29:08.471469 9424 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.221.188:35943
I20260810 22:29:08.479902 9593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:29:08.480077 9593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } }
I20260810 22:29:08.481144 9593 leader_election.cc:288] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cde8e631773940d9a966d0acdfad3aec (127.1.221.190:40671), 1c7324d97a184a1ba4e86eb31802017e (127.1.221.189:45551)
I20260810 22:29:08.482003 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 8861
I20260810 22:29:08.482165 9468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0da636ffd07b4335b5158a89e987d75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7324d97a184a1ba4e86eb31802017e" is_pre_election: true
I20260810 22:29:08.482362 9468 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 LEARNER]: Handling vote request from an unknown peer 0da636ffd07b4335b5158a89e987d75e
I20260810 22:29:08.482506 9468 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 0da636ffd07b4335b5158a89e987d75e for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260810 22:29:08.483165 8802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0da636ffd07b4335b5158a89e987d75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cde8e631773940d9a966d0acdfad3aec" is_pre_election: true
I20260810 22:29:08.484150 9530 leader_election.cc:302] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0da636ffd07b4335b5158a89e987d75e; no voters: 1c7324d97a184a1ba4e86eb31802017e, cde8e631773940d9a966d0acdfad3aec
I20260810 22:29:08.496277 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.474044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.1.221.190:40671)
0810 22:29:08.474179 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.474184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.474818 (+ 634us) server_negotiation.cc:407] Connection header received
0810 22:29:08.475025 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.475030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.475090 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.475145 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.476136 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.476673 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.478020 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.478371 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.478440 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.492293 (+ 13853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.492298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.492301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.492336 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.494299 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.494840 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.494844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.494846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.494936 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.495249 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.495253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.495255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.495490 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.495562 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.495948 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.496085 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:29:08.497032 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.474170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:35580)
0810 22:29:08.474715 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.474734 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.474898 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.475325 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.475328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.475684 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.475985 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.475994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.476802 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.476805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.477870 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.477884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.478620 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.478624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.478727 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.479304 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.479324 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.492057 (+ 12733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.494435 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.494438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.494440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.494717 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.495048 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.495051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.495053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.495148 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.495638 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.495643 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.495745 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.496665 (+ 920us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.496836 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":125,"spinlock_wait_cycles":33920,"thread_start_us":124,"threads_started":1}
I20260810 22:29:08.498888 9593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:08.502524 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.474750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.1.221.189:45551)
0810 22:29:08.474859 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.474863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.477849 (+ 2986us) server_negotiation.cc:407] Connection header received
0810 22:29:08.478029 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.478032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.478087 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.478152 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.478797 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.479613 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.480914 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.481279 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.481354 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.494162 (+ 12808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.494170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.494177 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.494222 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.496196 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.500950 (+ 4754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.500955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.500957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.501076 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.501485 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.501489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.501491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.501739 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.501824 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.502233 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.502369 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 22:29:08.502597 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.474209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.1.221.188:35943)
0810 22:29:08.474326 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.474331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.475094 (+ 763us) server_negotiation.cc:407] Connection header received
0810 22:29:08.475162 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.475165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.475211 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.475298 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.476132 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.476895 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.478050 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.478365 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.478437 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.496569 (+ 18132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.496575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.496581 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.496613 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.499901 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.500973 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.500979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.500984 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.501071 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.501491 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.501497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.501502 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.501750 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.501827 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.502212 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.502360 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 22:29:08.503026 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.474676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:45551 (local address 127.0.0.1:46532)
0810 22:29:08.477759 (+ 3083us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.477774 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.477917 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.478297 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.478300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.478531 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.478654 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.478662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.479931 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.479934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.480778 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.480789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.481505 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.481508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.481626 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.482047 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.482062 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.493925 (+ 11863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.500558 (+ 6633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.500564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.500568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.500827 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.501261 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.501264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.501266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.501374 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.501879 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.501882 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.502036 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.502705 (+ 669us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.502850 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3016,"mutex_wait_us":29,"thread_start_us":87,"threads_started":1}
I20260810 22:29:08.503679 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.474292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35943 (local address 127.0.0.1:58540)
0810 22:29:08.474997 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.475010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.475091 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.475326 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.475328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.475684 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.475985 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.475994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.477006 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.477009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.477950 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.477959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.479425 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.479432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.479583 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.480718 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.480732 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.492553 (+ 11821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.500159 (+ 7606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.500165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.500170 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.500474 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.501277 (+ 803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.501283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.501288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.501390 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.501888 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.501893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.502045 (+ 152us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.503331 (+ 1286us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.503485 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":120,"spinlock_wait_cycles":1280,"thread_start_us":116,"threads_started":1}
I20260810 22:29:08.503827 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:33893
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:40671
--webserver_port=33781
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.221.190:40671,127.1.221.189:45551
--tserver_master_addrs=127.1.221.190:40671,127.1.221.189:45551,127.1.221.188:35943 with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:29:08.523590 9593 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400946, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.190@KRBTEST.COM for oryx/127.1.221.188@KRBTEST.COM
I20260810 22:29:08.533805 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.524575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58556 (local address 127.1.221.188:35943)
0810 22:29:08.524682 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.524687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.524700 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:29:08.524776 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.524779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.524830 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.524902 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.525538 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.526324 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.527437 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.527736 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.527847 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.530290 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.530295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.530298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.530326 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.531975 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.532471 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.532474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.532476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.532566 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.532867 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.532873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.532875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.533095 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.533197 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.533493 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.533647 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:29:08.533782 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.524074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35943 (local address 127.0.0.1:58556)
0810 22:29:08.524341 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.524356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.524442 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.525040 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.525043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.525275 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.525403 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.525410 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.526451 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.526454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.527305 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.527317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.527864 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.527867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.527969 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.528511 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.528526 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.530133 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.532098 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.532101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.532103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.532359 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.532669 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.532672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.532675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.532767 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.533201 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.533205 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.533311 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.533419 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.533591 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260810 22:29:08.535315 8792 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.221.188:35943: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0da636ffd07b4335b5158a89e987d75e"
member_type: NON_VOTER
last_known_addr {
host: "127.1.221.188"
port: 35943
}
attrs {
promote: true
}
}
dest_uuid: "cde8e631773940d9a966d0acdfad3aec"
cas_config_opid_index: 5
I20260810 22:29:08.535931 8792 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } }
I20260810 22:29:08.536988 9409 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0da636ffd07b4335b5158a89e987d75e (127.1.221.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } } }
I20260810 22:29:08.538529 9409 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.221.188:35943
I20260810 22:29:08.538712 9409 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } } }
I20260810 22:29:08.538808 9409 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:08.539861 9560 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 FOLLOWER]: Refusing update from remote peer cde8e631773940d9a966d0acdfad3aec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 22:29:08.540515 9410 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:08.541445 9424 master.cc:537] Master@127.1.221.188:35943 shutting down...
I20260810 22:29:08.544086 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.537878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:45551 (local address 127.0.0.1:46548)
0810 22:29:08.538472 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.538486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.538558 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.539539 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.539543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.539562 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:08.539646 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.539651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.540822 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.540826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.542191 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.542204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.543624 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.543627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.543723 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.543775 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.543848 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.543946 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519}
I20260810 22:29:08.544588 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.539014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.1.221.189:45551)
0810 22:29:08.539149 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.539154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.539170 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:29:08.539225 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.539228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.539299 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.539390 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:08.539777 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.540561 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.542584 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.543255 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.543352 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.543519 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.544373 (+ 854us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.544474 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 22:29:08.544334 8775 consensus_peers.cc:487] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec -> Peer 0da636ffd07b4335b5158a89e987d75e (127.1.221.188:35943): Couldn't send request to peer 0da636ffd07b4335b5158a89e987d75e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:29:08.545887 9468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 LEARNER]: Refusing update from remote peer cde8e631773940d9a966d0acdfad3aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 22:29:08.546458 9410 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 22:29:08.548460 9468 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0da636ffd07b4335b5158a89e987d75e (127.1.221.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } } }
I20260810 22:29:08.550593 9424 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:08.550868 9424 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:08.550956 9424 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: stopping tablet replica
I20260810 22:29:08.550876 9500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde8e631773940d9a966d0acdfad3aec. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } } }
I20260810 22:29:08.550992 9500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: This master's current role is: LEARNER
I20260810 22:29:08.551106 9500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: NON_VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: true } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } } }
I20260810 22:29:08.551179 9500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: This master's current role is: LEARNER
I20260810 22:29:08.552775 9410 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec: attempting to promote NON_VOTER 1c7324d97a184a1ba4e86eb31802017e to VOTER
I20260810 22:29:08.553239 9410 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } }
W20260810 22:29:08.555131 8775 consensus_peers.cc:487] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec -> Peer 0da636ffd07b4335b5158a89e987d75e (127.1.221.188:35943): Couldn't send request to peer 0da636ffd07b4335b5158a89e987d75e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:29:08.555162 9468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 LEARNER]: Refusing update from remote peer cde8e631773940d9a966d0acdfad3aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 22:29:08.555538 9409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 22:29:08.558063 9409 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1c7324d97a184a1ba4e86eb31802017e (127.1.221.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } } }
I20260810 22:29:08.559754 9409 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } } }
I20260810 22:29:08.559898 9409 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:08.560228 9410 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } } }
I20260810 22:29:08.560333 9410 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:08.560753 9468 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1c7324d97a184a1ba4e86eb31802017e (127.1.221.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } } }
I20260810 22:29:08.562407 9616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } } }
I20260810 22:29:08.562541 9616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:29:08.569490 9424 master.cc:552] Master@127.1.221.188:35943 shutdown complete.
I20260810 22:29:08.569564 9424 master_runner.cc:305] Clearing existing system tablet
I20260810 22:29:08.569897 9424 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:08.569959 9424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:08.571360 9424 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 22:29:08.572399 9621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:08.572623 9424 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:08.572770 9424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 22:29:08.572865 9424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/wal
uuid: "0da636ffd07b4335b5158a89e987d75e"
format_stamp: "Formatted at 2026-08-10 22:29:08 on dist-test-slave-njxd"
server_key: "ccaf7efb4defa1b0826b66c589213356"
server_key_iv: "a7e0d66e3df8dc886b9e85520d79efde"
server_key_version: "encryptionkey@0"
I20260810 22:29:08.574937 9424 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:08.577309 9424 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 22:29:08.577402 9424 log.cc:1196] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 22:29:08.577700 9424 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: Deleting consensus metadata
I20260810 22:29:08.577868 9424 master_runner.cc:315] Copying system tablet from 127.1.221.190:40671
I20260810 22:29:08.579115 9424 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: tablet copy: Beginning tablet copy session from remote peer at address 127.1.221.190:40671
I20260810 22:29:08.594641 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.581928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:35590)
0810 22:29:08.585777 (+ 3849us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.585795 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.585940 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.586403 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.586409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.586636 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.586746 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.586754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.587824 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.587828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.588769 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.588781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.589510 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.589514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.589656 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.590261 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.590289 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.590747 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.592879 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.592883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.592885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.593139 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.593456 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.593462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.593464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.593597 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.594031 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.594034 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.594142 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.594257 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.594413 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3765,"thread_start_us":98,"threads_started":1}
I20260810 22:29:08.594688 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.582044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.1.221.190:40671)
0810 22:29:08.582197 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.582202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.585917 (+ 3715us) server_negotiation.cc:407] Connection header received
0810 22:29:08.586051 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.586054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.586113 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.586207 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.586883 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.587706 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.588908 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.589240 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.589353 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.590909 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.590913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.590918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.590948 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.592751 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.593246 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.593250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.593252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.593346 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.593684 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.593687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.593689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.593904 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.594022 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.594350 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.594473 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:29:08.595364 8814 tablet_copy_service.cc:140] P cde8e631773940d9a966d0acdfad3aec: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0da636ffd07b4335b5158a89e987d75e ({username='oryx', principal='oryx/127.1.221.188@KRBTEST.COM'} at 127.0.0.1:35590)
I20260810 22:29:08.595479 8814 tablet_copy_service.cc:161] P cde8e631773940d9a966d0acdfad3aec: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0da636ffd07b4335b5158a89e987d75e at {username='oryx', principal='oryx/127.1.221.188@KRBTEST.COM'} at 127.0.0.1:35590: session id = 0da636ffd07b4335b5158a89e987d75e-00000000000000000000000000000000
I20260810 22:29:08.596658 8814 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:29:08.599068 9424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 22:29:08.601734 9424 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: tablet copy: Starting download of 0 data blocks...
I20260810 22:29:08.601935 9424 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: tablet copy: Starting download of 1 WAL segments...
I20260810 22:29:08.603401 9424 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:29:08.605938 8814 tablet_copy_service.cc:342] P cde8e631773940d9a966d0acdfad3aec: Request end of tablet copy session 0da636ffd07b4335b5158a89e987d75e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.221.188@KRBTEST.COM'} at 127.0.0.1:35590
I20260810 22:29:08.606025 8814 tablet_copy_service.cc:433] P cde8e631773940d9a966d0acdfad3aec: ending tablet copy session 0da636ffd07b4335b5158a89e987d75e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0da636ffd07b4335b5158a89e987d75e
I20260810 22:29:08.616703 9424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:08.618474 9632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:08.618878 9424 server_base.cc:1034] running on GCE node
W20260810 22:29:08.618824 9630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:08.621994 9629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:08.622251 9424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:08.622603 9424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:08.623772 9424 hybrid_clock.cc:648] HybridClock initialized: now 1786400948623751 us; error 51 us; skew 500 ppm
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:08.625473 9424 init.cc:377] Logged in from keytab as oryx/127.1.221.188@KRBTEST.COM (short username oryx)
I20260810 22:29:08.625972 9424 webserver.cc:460] Webserver started at http://127.1.221.188:34205/ using document root <none> and password file <none>
I20260810 22:29:08.626230 9424 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:08.626272 9424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:08.627563 9424 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 22:29:08.628636 9639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:08.628751 9148 heartbeater.cc:503] Master 127.1.221.190:40671 was elected leader, sending a full tablet report...
I20260810 22:29:08.629110 9424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 22:29:08.629189 9424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/wal
uuid: "0da636ffd07b4335b5158a89e987d75e"
format_stamp: "Formatted at 2026-08-10 22:29:08 on dist-test-slave-njxd"
server_key: "ccaf7efb4defa1b0826b66c589213356"
server_key_iv: "a7e0d66e3df8dc886b9e85520d79efde"
server_key_version: "encryptionkey@0"
I20260810 22:29:08.629261 9424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260810 22:29:08.646807 9614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:08.647094 9614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:08.647189 9614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:08.651473 9614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:08.651568 9614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:08.651674 9614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:29:08.657280 9614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:33893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=33781
--tserver_master_addrs=127.1.221.190:40671,127.1.221.189:45551,127.1.221.188:35943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:08.658481 9614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:08.659535 9614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:08.670159 9651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:08.670169 9650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:08.670382 9654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:08.671149 9614 server_base.cc:1034] running on GCE node
I20260810 22:29:08.671559 9614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:08.672403 9614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:08.673086 9424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:08.673616 9614 hybrid_clock.cc:648] HybridClock initialized: now 1786400948673571 us; error 54 us; skew 500 ppm
I20260810 22:29:08.675014 9424 kserver.cc:163] Server-wide thread pool size limit: 3276
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:08.676857 9614 init.cc:377] Logged in from keytab as oryx/127.1.221.129@KRBTEST.COM (short username oryx)
I20260810 22:29:08.678066 9614 webserver.cc:460] Webserver started at http://127.1.221.129:33781/ using document root <none> and password file <none>
I20260810 22:29:08.678876 9614 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:08.678998 9614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:08.680193 9424 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.188:35943
I20260810 22:29:08.681018 9424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/master-2/data/info.pb
I20260810 22:29:08.681219 9711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.188:35943 every 8 connection(s)
I20260810 22:29:08.682664 9712 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:29:08.683797 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: Bootstrap starting.
I20260810 22:29:08.684015 9614 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:29:08.686262 9715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:08.687565 9614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:29:08.687716 9614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "a4013ce0110a486eb0e2e01623774c2c"
format_stamp: "Formatted at 2026-08-10 22:29:07 on dist-test-slave-njxd"
server_key: "9aebbd169c8ecb1bf4d1897a755aaaa7"
server_key_iv: "e431fd319943de59ddd416b178fce116"
server_key_version: "encryptionkey@0"
I20260810 22:29:08.688269 9614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:08.693249 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:29:08.693535 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: Bootstrap complete.
I20260810 22:29:08.694031 9712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } }
I20260810 22:29:08.694141 9712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0da636ffd07b4335b5158a89e987d75e, State: Initialized, Role: LEARNER
I20260810 22:29:08.694254 9712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } }
I20260810 22:29:08.694617 9721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } } }
I20260810 22:29:08.694734 9721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: This master's current role is: LEARNER
I20260810 22:29:08.694777 9712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:08.697587 9733 catalog_manager.cc:1259] Loaded cluster ID: a1199cc48bca4f2c86716b1fe896bbf4
I20260810 22:29:08.697659 9733 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: loading cluster ID for follower catalog manager: success
I20260810 22:29:08.699203 9733 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: acquiring CA information for follower catalog manager: success
I20260810 22:29:08.700381 9733 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:29:08.720698 9614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:08.723758 9614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:08.723984 9614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:08.724635 9614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:08.725643 9614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:08.725690 9614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:08.725762 9614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:08.725793 9614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:08.735287 9614 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:33893
I20260810 22:29:08.735414 9852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:33893 every 8 connection(s)
I20260810 22:29:08.736416 9614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:29:08.740111 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 9614
I20260810 22:29:08.740352 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 9009
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.188@KRBTEST.COM
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
I20260810 22:29:08.754487 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.738368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:40679 (local address 127.1.221.188:35943)
0810 22:29:08.738863 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.738876 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.739469 (+ 593us) server_negotiation.cc:407] Connection header received
0810 22:29:08.740949 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.740954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.741018 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.741078 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.742702 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.743418 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.745178 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.745541 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.745659 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.749869 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.749876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.749882 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.749920 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.751843 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.752476 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.752480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.752482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.752574 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.752975 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.752978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.752979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.753220 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.753297 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.754097 (+ 800us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.754274 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":86,"threads_started":1}
I20260810 22:29:08.755168 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.740139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:60523 (local address 127.1.221.189:45551)
0810 22:29:08.740266 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.740270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.740287 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:29:08.741452 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.741456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.741512 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.741990 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.742748 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.743771 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.745394 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.745932 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.746048 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.750136 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.750144 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.750150 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.750191 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.752567 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.753120 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.753126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.753131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.753226 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.753627 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.753633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.753638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.753888 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.753967 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.754783 (+ 816us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.754926 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260810 22:29:08.757476 9859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.738762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35943 (local address 127.1.221.129:40679)
0810 22:29:08.739315 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.739368 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.740505 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.741407 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.741417 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.741953 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.742543 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.742557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.743578 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.743583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.745040 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.745054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.745868 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.745874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.746100 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.746923 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.746960 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.749680 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.751957 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.751963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.751982 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.752356 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.752682 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.752685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.752687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.752873 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.753387 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.753394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.753788 (+ 394us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.754932 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.756738 (+ 1806us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":1098,"spinlock_wait_cycles":15616,"thread_start_us":569,"threads_started":1}
I20260810 22:29:08.757967 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:39725
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:40671
--webserver_port=39553
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.221.190:40671,127.1.221.189:45551
--tserver_master_addrs=127.1.221.190:40671,127.1.221.189:45551,127.1.221.188:35943 with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:29:08.757489 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.739856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:45551 (local address 127.1.221.129:60523)
0810 22:29:08.740110 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.740135 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.740833 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.742047 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.742053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.742322 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.742555 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.742571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.743914 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.743920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.745217 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.745233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.746271 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.746279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.746431 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.746922 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.746972 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.749952 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.752716 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.752722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.752727 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.752996 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.753382 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.753388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.753393 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.753494 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.754049 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.754054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.754458 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.755401 (+ 943us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.755631 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":27,"spinlock_wait_cycles":11904,"thread_start_us":79,"threads_started":1}
I20260810 22:29:08.758973 9853 heartbeater.cc:347] Connected to a master server at 127.1.221.188:35943
I20260810 22:29:08.759231 9853 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:08.759384 9855 heartbeater.cc:347] Connected to a master server at 127.1.221.189:45551
I20260810 22:29:08.759500 9855 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:08.759933 9855 heartbeater.cc:511] Master 127.1.221.189:45551 requested a full tablet report, sending...
I20260810 22:29:08.759933 9853 heartbeater.cc:511] Master 127.1.221.188:35943 requested a full tablet report, sending...
I20260810 22:29:08.760663 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.738395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:49459 (local address 127.1.221.190:40671)
0810 22:29:08.738528 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.738532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.739677 (+ 1145us) server_negotiation.cc:407] Connection header received
0810 22:29:08.748753 (+ 9076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.748756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.748846 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.748941 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.750913 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.751751 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.752929 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.753265 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.753343 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.756134 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.756138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.756140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.756169 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.757957 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.759112 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.759129 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.759133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.759227 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.759560 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.759566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.759570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.759820 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.759900 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.760321 (+ 421us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.760465 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:29:08.761230 9860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.738762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.129:49459)
0810 22:29:08.739564 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.739578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.740679 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.749038 (+ 8359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.749044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.749472 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.749693 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.749708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.751906 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.751911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.752792 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.752800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.753507 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.753510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.753914 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.754387 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.754401 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.755974 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.758089 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.758095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.758100 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.758981 (+ 881us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.759351 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.759356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.759361 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.759456 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.759986 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.759991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.760099 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.760818 (+ 719us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.760986 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":30,"spinlock_wait_cycles":1280,"thread_start_us":158,"threads_started":1}
I20260810 22:29:08.761605 9671 ts_manager.cc:194] Registered new tserver with Master: a4013ce0110a486eb0e2e01623774c2c (127.1.221.129:33893)
I20260810 22:29:08.761565 9455 ts_manager.cc:194] Registered new tserver with Master: a4013ce0110a486eb0e2e01623774c2c (127.1.221.129:33893)
I20260810 22:29:08.761842 9854 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40671
I20260810 22:29:08.761922 9854 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:08.762131 9854 heartbeater.cc:511] Master 127.1.221.190:40671 requested a full tablet report, sending...
I20260810 22:29:08.763013 8792 ts_manager.cc:194] Re-registered known tserver with Master: a4013ce0110a486eb0e2e01623774c2c (127.1.221.129:33893)
I20260810 22:29:08.763891 8792 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:49459
I20260810 22:29:08.849927 9295 heartbeater.cc:503] Master 127.1.221.190:40671 was elected leader, sending a full tablet report...
W20260810 22:29:08.877135 9862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:08.877383 9862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:08.877462 9862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:08.881557 9862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:08.881666 9862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:08.881779 9862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:29:08.887141 9862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:39725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=39553
--tserver_master_addrs=127.1.221.190:40671,127.1.221.189:45551,127.1.221.188:35943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:08.888211 9862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:08.889281 9862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:08.899276 9870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:08.899262 9868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:08.899293 9867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:08.899507 9862 server_base.cc:1034] running on GCE node
I20260810 22:29:08.900022 9862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:08.900765 9862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:08.901924 9862 hybrid_clock.cc:648] HybridClock initialized: now 1786400948901893 us; error 43 us; skew 500 ppm
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:08.904546 9862 init.cc:377] Logged in from keytab as oryx/127.1.221.130@KRBTEST.COM (short username oryx)
I20260810 22:29:08.905393 9862 webserver.cc:460] Webserver started at http://127.1.221.130:39553/ using document root <none> and password file <none>
I20260810 22:29:08.906004 9862 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:08.906104 9862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:08.909984 9862 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:29:08.912222 9877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:08.913638 9862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:29:08.913779 9862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "9e00f7e5f7ca459a866c8b78fc0df48a"
format_stamp: "Formatted at 2026-08-10 22:29:07 on dist-test-slave-njxd"
server_key: "6cc9d313407c7427e1faa5ccd62544b9"
server_key_iv: "b308281f9d80d1f9dc8fc93f2bd958b8"
server_key_version: "encryptionkey@0"
I20260810 22:29:08.914372 9862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:08.928210 9862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:08.931774 9862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:08.931999 9862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:08.932672 9862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:08.933660 9862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:08.933738 9862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:08.933812 9862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:08.933844 9862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:08.942788 9862 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:39725
I20260810 22:29:08.942821 9999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:39725 every 8 connection(s)
I20260810 22:29:08.943811 9862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:29:08.947438 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 9862
I20260810 22:29:08.947625 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 9153
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.188@KRBTEST.COM
Aug 10 22:29:08 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400948, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260810 22:29:08.964597 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:43113
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:40671
--webserver_port=34679
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.221.190:40671,127.1.221.189:45551
--tserver_master_addrs=127.1.221.190:40671,127.1.221.189:45551,127.1.221.188:35943 with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:29:08.967021 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.947091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:55337 (local address 127.1.221.189:45551)
0810 22:29:08.947220 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.947225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.951502 (+ 4277us) server_negotiation.cc:407] Connection header received
0810 22:29:08.951846 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.951850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.951908 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.951966 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.954420 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.955169 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.957692 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.958047 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.958126 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.961736 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.961742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.961748 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.961784 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.964012 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.964847 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.964850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.964852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.964964 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.965523 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.965527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.965529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.965793 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.965881 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.966713 (+ 832us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.966849 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:29:08.967021 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.945634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:48033 (local address 127.1.221.188:35943)
0810 22:29:08.945794 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.945799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.946870 (+ 1071us) server_negotiation.cc:407] Connection header received
0810 22:29:08.951848 (+ 4978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.951851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.951908 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.951966 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.956872 (+ 4906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.957725 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.958857 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.959261 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.959362 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.962011 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.962020 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.962025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.962060 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.963966 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.964855 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.964859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.964861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.965085 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.965523 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.965527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.965529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.965783 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.965881 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.966713 (+ 832us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.966861 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 22:29:08.968135 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.945677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:51235 (local address 127.1.221.190:40671)
0810 22:29:08.945825 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:08.945830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:08.947233 (+ 1403us) server_negotiation.cc:407] Connection header received
0810 22:29:08.957666 (+ 10433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:08.957671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:08.957734 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:08.957823 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:08.958452 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.959184 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.960294 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.960626 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.960701 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.963342 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:08.963346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:08.963348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:08.963380 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:08.965258 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.966060 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.966066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.966071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.966167 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:08.966557 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:08.966562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:08.966565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:08.966822 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:08.966914 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:08.967807 (+ 893us) server_negotiation.cc:299] Negotiation successful
0810 22:29:08.967951 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 22:29:08.968843 10004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.946048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35943 (local address 127.1.221.130:48033)
0810 22:29:08.946648 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.946716 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.947694 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.952423 (+ 4729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.952434 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.953086 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.953805 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.953829 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.957845 (+ 4016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.957848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.958701 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.958713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.959542 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.959546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.959682 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.960221 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.960242 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.961843 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.964116 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.964124 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.964144 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.964725 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.965185 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.965188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.965190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.965400 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.965959 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.965966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.966397 (+ 431us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.967208 (+ 811us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.967708 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":159,"spinlock_wait_cycles":14848,"thread_start_us":696,"threads_started":1}
I20260810 22:29:08.969136 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.946967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:45551 (local address 127.1.221.130:55337)
0810 22:29:08.951404 (+ 4437us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.951420 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.951574 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.952423 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.952434 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.954134 (+ 1700us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.954273 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.954283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.955364 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.955370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.956702 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.956714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.958352 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.958356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.958549 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.959269 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.959292 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.961503 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.964130 (+ 2627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.964132 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.964144 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.964717 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.965218 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.965224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.965230 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.965400 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.965959 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.965966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.966397 (+ 431us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.967244 (+ 847us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.967655 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":88,"spinlock_wait_cycles":11136,"thread_start_us":79,"threads_started":1}
I20260810 22:29:08.969899 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:08.946048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.130:51235)
0810 22:29:08.947136 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:08.947149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:08.957482 (+ 10333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:08.957952 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:08.957955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:08.958155 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:08.958291 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.958299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.959347 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.959351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.960136 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:08.960147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:08.960991 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:08.960996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:08.961084 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:08.961667 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:08.961682 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:08.963158 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:08.965657 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.965663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.965668 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.965935 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.966304 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:08.966309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:08.966313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:08.966432 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:08.966978 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:08.966982 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:08.967580 (+ 598us) client_negotiation.cc:769] Sending connection context
0810 22:29:08.968584 (+ 1004us) client_negotiation.cc:240] Negotiation successful
0810 22:29:08.968768 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"mutex_wait_us":42,"spinlock_wait_cycles":768,"thread_start_us":124,"threads_started":1}
I20260810 22:29:08.970414 10000 heartbeater.cc:347] Connected to a master server at 127.1.221.188:35943
I20260810 22:29:08.970496 10001 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40671
I20260810 22:29:08.970675 10001 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:08.970675 10000 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:08.970701 10002 heartbeater.cc:347] Connected to a master server at 127.1.221.189:45551
I20260810 22:29:08.971231 10002 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:08.971414 10000 heartbeater.cc:511] Master 127.1.221.188:35943 requested a full tablet report, sending...
I20260810 22:29:08.971413 10002 heartbeater.cc:511] Master 127.1.221.189:45551 requested a full tablet report, sending...
I20260810 22:29:08.972082 10001 heartbeater.cc:511] Master 127.1.221.190:40671 requested a full tablet report, sending...
I20260810 22:29:08.972762 9455 ts_manager.cc:194] Registered new tserver with Master: 9e00f7e5f7ca459a866c8b78fc0df48a (127.1.221.130:39725)
I20260810 22:29:08.972937 8792 ts_manager.cc:194] Re-registered known tserver with Master: 9e00f7e5f7ca459a866c8b78fc0df48a (127.1.221.130:39725)
I20260810 22:29:08.972987 9671 ts_manager.cc:194] Registered new tserver with Master: 9e00f7e5f7ca459a866c8b78fc0df48a (127.1.221.130:39725)
I20260810 22:29:08.973860 8792 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:51235
I20260810 22:29:09.070737 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.066464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35943 (local address 127.0.0.1:58566)
0810 22:29:09.066723 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:09.066737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:09.066861 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:09.067181 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:09.067183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:09.067198 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:09.067294 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.067299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.068340 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.068342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.069100 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.069108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.070118 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.070125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.070296 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.070337 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 22:29:09.070430 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 22:29:09.070528 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
I20260810 22:29:09.070792 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.066573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58566 (local address 127.1.221.188:35943)
0810 22:29:09.066718 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.066724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.066838 (+ 114us) server_negotiation.cc:407] Connection header received
0810 22:29:09.066968 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.066971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.067043 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.067154 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:09.067399 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.068221 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.069245 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.069933 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.070042 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.070206 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.070465 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.070603 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:29:09.071765 9682 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 22:29:09.072306 9721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde8e631773940d9a966d0acdfad3aec. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } } }
I20260810 22:29:09.072470 9721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: This master's current role is: LEARNER
W20260810 22:29:09.090175 10007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:09.090436 10007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:09.090480 10007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:09.094614 10007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:09.094671 10007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:09.094729 10007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:29:09.100387 10007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:43113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=34679
--tserver_master_addrs=127.1.221.190:40671,127.1.221.189:45551,127.1.221.188:35943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:09.101415 10007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:09.102474 10007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:09.112366 10013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:09.112398 10016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:09.112366 10014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:09.112566 10007 server_base.cc:1034] running on GCE node
I20260810 22:29:09.113185 10007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:09.113965 10007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:09.115160 10007 hybrid_clock.cc:648] HybridClock initialized: now 1786400949115142 us; error 31 us; skew 500 ppm
Aug 10 22:29:09 dist-test-slave-njxd krb5kdc[8743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400949, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:09.118034 10007 init.cc:377] Logged in from keytab as oryx/127.1.221.131@KRBTEST.COM (short username oryx)
I20260810 22:29:09.118906 10007 webserver.cc:460] Webserver started at http://127.1.221.131:34679/ using document root <none> and password file <none>
I20260810 22:29:09.119445 10007 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:09.119518 10007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:09.123528 10007 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:29:09.125638 10024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:09.126924 10007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:29:09.127031 10007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "3aa8d37a0d844500aa9fe801dd62efff"
format_stamp: "Formatted at 2026-08-10 22:29:07 on dist-test-slave-njxd"
server_key: "e81825707d40a2a5223fe30d0c27a187"
server_key_iv: "f44d53059a7c1f0da143a8f91999579b"
server_key_version: "encryptionkey@0"
I20260810 22:29:09.127558 10007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:09.144284 10007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:09.147336 10007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:09.147503 10007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:09.148069 10007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:09.148921 10007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:09.148968 10007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:09.149010 10007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:09.149032 10007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:09.158416 10007 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:43113
I20260810 22:29:09.158439 10148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:43113 every 8 connection(s)
I20260810 22:29:09.159468 10007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:29:09.164363 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 10007
Aug 10 22:29:09 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400949, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.189@KRBTEST.COM
Aug 10 22:29:09 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400949, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
Aug 10 22:29:09 dist-test-slave-njxd krb5kdc[8743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400949, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.188@KRBTEST.COM
I20260810 22:29:09.179839 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.163351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:55175 (local address 127.1.221.189:45551)
0810 22:29:09.163481 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.163485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.163623 (+ 138us) server_negotiation.cc:407] Connection header received
0810 22:29:09.165057 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.165063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.165126 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.165185 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:09.167706 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.168490 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.169625 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.170018 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.170134 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.173458 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:09.173462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:09.173465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:09.173491 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:09.175914 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.176824 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.176830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.176835 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.176933 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.178074 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.178081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.178085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.178356 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:09.178478 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.179503 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.179646 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:29:09.180106 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.161446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:34179 (local address 127.1.221.188:35943)
0810 22:29:09.161603 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.161607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.162874 (+ 1267us) server_negotiation.cc:407] Connection header received
0810 22:29:09.164173 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.164177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.164236 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.164302 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:09.166413 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.167089 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.169471 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.169822 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.169910 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.175124 (+ 5214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:09.175127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:09.175130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:09.175153 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:09.176870 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.178592 (+ 1722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.178595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.178597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.178692 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.179050 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.179054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.179055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.179299 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:09.179389 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.179823 (+ 434us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.179944 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 22:29:09.181057 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.161629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:57507 (local address 127.1.221.190:40671)
0810 22:29:09.161768 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.161772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.162875 (+ 1103us) server_negotiation.cc:407] Connection header received
0810 22:29:09.164173 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.164177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.164236 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.164292 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:09.166432 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.167187 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.168917 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.169244 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.169338 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.174600 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:09.174605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:09.174610 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:09.174646 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:09.178394 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.179311 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.179315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.179317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.179408 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.179732 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.179736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.179738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.179961 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:09.180050 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.180804 (+ 754us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.180922 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:29:09.181470 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.162080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35943 (local address 127.1.221.131:34179)
0810 22:29:09.162691 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:09.162757 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:09.163922 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:09.164720 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:09.164729 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:09.165783 (+ 1054us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:09.166259 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.166271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.167295 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.167300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.169343 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.169352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.170100 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.170105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.170207 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.173502 (+ 3295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:09.173516 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:09.175009 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:09.177335 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:09.177338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:09.177341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:09.178484 (+ 1143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:09.178811 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:09.178817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:09.178822 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:09.178919 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:09.179492 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:09.179495 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:09.179597 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 22:29:09.180309 (+ 712us) client_negotiation.cc:240] Negotiation successful
0810 22:29:09.180535 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":1057,"spinlock_wait_cycles":63616,"thread_start_us":135,"threads_started":1}
I20260810 22:29:09.181771 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.163244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:45551 (local address 127.1.221.131:55175)
0810 22:29:09.163511 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:09.163525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:09.163927 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:09.165380 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:09.165386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:09.165596 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:09.167546 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.167558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.168599 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.168604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.169457 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.169467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.170147 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.170151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.170235 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.170794 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:09.170809 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:09.173266 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:09.176286 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:09.176296 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:09.176314 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:09.176695 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:09.177719 (+ 1024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:09.177724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:09.177730 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:09.177943 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:09.178557 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:09.178567 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:09.178915 (+ 348us) client_negotiation.cc:769] Sending connection context
0810 22:29:09.179144 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 22:29:09.180191 (+ 1047us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":77,"thread_start_us":101,"threads_started":1}
I20260810 22:29:09.183121 10152 heartbeater.cc:347] Connected to a master server at 127.1.221.189:45551
I20260810 22:29:09.183409 10152 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:09.184031 10152 heartbeater.cc:511] Master 127.1.221.189:45551 requested a full tablet report, sending...
I20260810 22:29:09.183104 10149 heartbeater.cc:347] Connected to a master server at 127.1.221.188:35943
I20260810 22:29:09.184695 10149 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:09.184860 10149 heartbeater.cc:511] Master 127.1.221.188:35943 requested a full tablet report, sending...
I20260810 22:29:09.185540 9455 ts_manager.cc:194] Registered new tserver with Master: 3aa8d37a0d844500aa9fe801dd62efff (127.1.221.131:43113)
I20260810 22:29:09.186411 9671 ts_manager.cc:194] Registered new tserver with Master: 3aa8d37a0d844500aa9fe801dd62efff (127.1.221.131:43113)
I20260810 22:29:09.186714 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.166901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.1.221.189:45551)
0810 22:29:09.171036 (+ 4135us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.171041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.171055 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:29:09.171087 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.171090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.171133 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.172154 (+ 1021us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:09.172270 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.173055 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.174942 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.175344 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.177103 (+ 1759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.179012 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:09.179019 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:09.179023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:09.179048 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:09.183859 (+ 4811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.184470 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.184473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.184475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.184566 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.185164 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.185167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.185168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.185519 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:09.185738 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.186391 (+ 653us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.186529 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4047,"thread_start_us":96,"threads_started":1}
I20260810 22:29:09.187448 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.162080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.131:57507)
0810 22:29:09.162691 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:09.162757 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:09.163922 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:09.164720 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:09.164729 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:09.165439 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:09.166248 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.166273 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.167347 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.167353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.168783 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.168794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.169502 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.169506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.169711 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.170530 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:09.170558 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:09.174432 (+ 3874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:09.178944 (+ 4512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:09.178947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:09.178949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:09.179201 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:09.179525 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:09.179528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:09.179530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:09.179623 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:09.180505 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:09.180508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:09.180596 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 22:29:09.181463 (+ 867us) client_negotiation.cc:240] Negotiation successful
0810 22:29:09.186355 (+ 4892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":181,"spinlock_wait_cycles":7808,"thread_start_us":158,"threads_started":1}
I20260810 22:29:09.187641 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.167184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35596 (local address 127.1.221.190:40671)
0810 22:29:09.167829 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.167833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.168070 (+ 237us) server_negotiation.cc:407] Connection header received
0810 22:29:09.168225 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.168228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.168280 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.168372 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:09.172272 (+ 3900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.173063 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.178310 (+ 5247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.178770 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.178875 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.181946 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:09.181950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:09.181953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:09.181978 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:09.184859 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.185554 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.185557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.185559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.185685 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.185989 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.185992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.185994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.186782 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:09.186922 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.187193 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.187453 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":103,"threads_started":1}
I20260810 22:29:09.188323 10151 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40671
I20260810 22:29:09.188410 10151 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:09.188598 10151 heartbeater.cc:511] Master 127.1.221.190:40671 requested a full tablet report, sending...
I20260810 22:29:09.189733 8792 ts_manager.cc:194] Re-registered known tserver with Master: 3aa8d37a0d844500aa9fe801dd62efff (127.1.221.131:43113)
I20260810 22:29:09.189934 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.167666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.1.221.188:35943)
0810 22:29:09.168340 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.168355 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.169009 (+ 654us) server_negotiation.cc:407] Connection header received
0810 22:29:09.169049 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.169052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.169095 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.169165 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:09.171076 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.171913 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.173694 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.174098 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.174183 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.176960 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:09.176967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:09.176973 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:09.177004 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:09.178820 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.188354 (+ 9534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.188358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.188360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.188530 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.188846 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.188849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.188851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.189102 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:09.189242 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.189515 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.189690 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":116,"threads_started":1}
I20260810 22:29:09.190440 8792 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:57507
I20260810 22:29:09.191236 8792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:29:09.194587 8792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:29:09.197342 9409 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: NON_VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: true } } }
I20260810 22:29:09.197501 9409 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:09.200740 9409 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec: attempting to promote NON_VOTER 0da636ffd07b4335b5158a89e987d75e to VOTER
I20260810 22:29:09.201017 9409 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } }
I20260810 22:29:09.202188 9682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 LEARNER]: Refusing update from remote peer cde8e631773940d9a966d0acdfad3aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 22:29:09.202596 9409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 22:29:09.204131 10172 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0da636ffd07b4335b5158a89e987d75e (127.1.221.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } } }
I20260810 22:29:09.204535 9682 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0da636ffd07b4335b5158a89e987d75e (127.1.221.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } } }
I20260810 22:29:09.205137 9468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 FOLLOWER]: Refusing update from remote peer cde8e631773940d9a966d0acdfad3aec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 22:29:09.205602 10173 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 22:29:09.205667 10172 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } } }
I20260810 22:29:09.205767 10172 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:09.205892 10172 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } } }
I20260810 22:29:09.205971 10172 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:09.206403 9721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } } }
I20260810 22:29:09.206516 9721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:29:09.206434 9468 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0da636ffd07b4335b5158a89e987d75e (127.1.221.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } } }
I20260810 22:29:09.208001 10184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } } }
I20260810 22:29:09.208163 10184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:29:09.208393 10173 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cde8e631773940d9a966d0acdfad3aec" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } } }
I20260810 22:29:09.208524 10173 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:09.216450 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.209923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:43113 (local address 127.0.0.1:45096)
0810 22:29:09.210600 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:09.210618 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:09.210711 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:09.211842 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:09.211845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:09.211876 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:09.212192 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.212200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.213699 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.213702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.214883 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.214894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.215988 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.215991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.216084 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.216142 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 22:29:09.216215 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 22:29:09.216312 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":233,"spinlock_wait_cycles":4736}
I20260810 22:29:09.216472 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.210908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.1.221.131:43113)
0810 22:29:09.211294 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.211309 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.211335 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:29:09.211422 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.211428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.211690 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.211966 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:09.212315 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.213506 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.215005 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.215868 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.215987 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.216156 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.216245 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.216343 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":121,"threads_started":1}
I20260810 22:29:09.216467 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.210466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:39725 (local address 127.0.0.1:44208)
0810 22:29:09.210718 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:09.210742 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:09.211279 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:09.211844 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:09.211846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:09.211878 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:09.212382 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.212400 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.213873 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.213879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.214797 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.214806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.216009 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.216012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.216131 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.216160 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 22:29:09.216216 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 22:29:09.216304 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":12,"spinlock_wait_cycles":8192,"thread_start_us":125,"threads_started":1}
I20260810 22:29:09.217305 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.211400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33893 (local address 127.0.0.1:34362)
0810 22:29:09.211644 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:09.211658 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:09.211740 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:09.212950 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:09.212954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:09.212975 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:09.213038 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.213042 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.214262 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.214265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.215622 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.215630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.216961 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.216963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.217024 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.217039 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 22:29:09.217129 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 22:29:09.217194 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":116,"threads_started":1}
I20260810 22:29:09.217471 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.211858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34362 (local address 127.1.221.129:33893)
0810 22:29:09.212233 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.212253 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.212273 (+ 20us) server_negotiation.cc:407] Connection header received
0810 22:29:09.212366 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.212371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.212655 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.212807 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:09.213153 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.214130 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.215756 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.216686 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.216787 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.216910 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.217162 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.217256 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":73,"threads_started":1}
I20260810 22:29:09.217334 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.210904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.1.221.130:39725)
0810 22:29:09.211294 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.211309 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.211335 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:29:09.211422 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.211428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.211661 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.212545 (+ 884us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:09.212694 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.213783 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.215004 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.215868 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.216925 (+ 1057us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.217063 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.217109 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.217195 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":115,"threads_started":1}
I20260810 22:29:09.219627 10075 tablet_service.cc:1467] Processing CreateTablet for tablet 29a53a3c25ca41e3aaf0917b15a62e9d (DEFAULT_TABLE table=test-table [id=3a9cc73e81b5422bac98834526a31c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:09.219962 9928 tablet_service.cc:1467] Processing CreateTablet for tablet 29a53a3c25ca41e3aaf0917b15a62e9d (DEFAULT_TABLE table=test-table [id=3a9cc73e81b5422bac98834526a31c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:09.220139 9780 tablet_service.cc:1467] Processing CreateTablet for tablet 29a53a3c25ca41e3aaf0917b15a62e9d (DEFAULT_TABLE table=test-table [id=3a9cc73e81b5422bac98834526a31c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:09.220764 10075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a53a3c25ca41e3aaf0917b15a62e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:09.220986 9928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a53a3c25ca41e3aaf0917b15a62e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:09.221101 9780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29a53a3c25ca41e3aaf0917b15a62e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:09.228477 10191 tablet_bootstrap.cc:492] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff: Bootstrap starting.
I20260810 22:29:09.229005 10190 tablet_bootstrap.cc:492] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a: Bootstrap starting.
I20260810 22:29:09.229919 10192 tablet_bootstrap.cc:492] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c: Bootstrap starting.
I20260810 22:29:09.231006 10190 tablet_bootstrap.cc:654] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:09.231047 10191 tablet_bootstrap.cc:654] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:09.231807 10190 log.cc:824] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:09.231880 10191 log.cc:824] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:09.232082 10192 tablet_bootstrap.cc:654] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:09.232861 10192 log.cc:824] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:09.233971 10190 tablet_bootstrap.cc:492] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a: No bootstrap required, opened a new log
I20260810 22:29:09.234110 10190 ts_tablet_manager.cc:1397] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:29:09.234266 10191 tablet_bootstrap.cc:492] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff: No bootstrap required, opened a new log
I20260810 22:29:09.234433 10191 ts_tablet_manager.cc:1397] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260810 22:29:09.234870 10192 tablet_bootstrap.cc:492] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c: No bootstrap required, opened a new log
I20260810 22:29:09.235023 10192 ts_tablet_manager.cc:1397] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:29:09.237823 10190 raft_consensus.cc:348] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } }
I20260810 22:29:09.238145 10190 raft_consensus.cc:374] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:09.238204 10190 raft_consensus.cc:729] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e00f7e5f7ca459a866c8b78fc0df48a, State: Initialized, Role: FOLLOWER
I20260810 22:29:09.238665 10190 consensus_queue.cc:260] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } }
I20260810 22:29:09.238993 10191 raft_consensus.cc:348] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } }
I20260810 22:29:09.239240 10191 raft_consensus.cc:374] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:09.239288 10191 raft_consensus.cc:729] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa8d37a0d844500aa9fe801dd62efff, State: Initialized, Role: FOLLOWER
I20260810 22:29:09.239454 10192 raft_consensus.cc:348] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } }
I20260810 22:29:09.239672 10192 raft_consensus.cc:374] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:09.239733 10192 raft_consensus.cc:729] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4013ce0110a486eb0e2e01623774c2c, State: Initialized, Role: FOLLOWER
I20260810 22:29:09.239728 10190 ts_tablet_manager.cc:1428] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:29:09.240026 10001 heartbeater.cc:503] Master 127.1.221.190:40671 was elected leader, sending a full tablet report...
I20260810 22:29:09.240175 10192 consensus_queue.cc:260] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } }
I20260810 22:29:09.240849 9854 heartbeater.cc:503] Master 127.1.221.190:40671 was elected leader, sending a full tablet report...
I20260810 22:29:09.241153 10192 ts_tablet_manager.cc:1428] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260810 22:29:09.244277 10191 consensus_queue.cc:260] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } }
I20260810 22:29:09.245397 10191 ts_tablet_manager.cc:1428] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff: Time spent starting tablet: real 0.011s user 0.004s sys 0.003s
I20260810 22:29:09.246506 10151 heartbeater.cc:503] Master 127.1.221.190:40671 was elected leader, sending a full tablet report...
I20260810 22:29:09.296018 10198 raft_consensus.cc:482] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:29:09.296329 10198 raft_consensus.cc:504] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } }
I20260810 22:29:09.297528 10198 leader_election.cc:288] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4013ce0110a486eb0e2e01623774c2c (127.1.221.129:33893), 9e00f7e5f7ca459a866c8b78fc0df48a (127.1.221.130:39725)
I20260810 22:29:09.302184 10186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.297931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:35561 (local address 127.1.221.130:39725)
0810 22:29:09.298082 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.298087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.298379 (+ 292us) server_negotiation.cc:407] Connection header received
0810 22:29:09.298466 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.298469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.298560 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.298658 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:09.299057 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.299695 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.301020 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.301656 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.301763 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.301828 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.301975 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.302052 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:29:09.302186 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.297897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:43729 (local address 127.1.221.129:33893)
0810 22:29:09.298023 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.298028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.298045 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:29:09.298210 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.298213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.298257 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.298326 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:09.298668 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.299373 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.300902 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.301530 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.301692 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.301758 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.301976 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.302053 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:29:09.302217 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.297851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33893 (local address 127.1.221.131:43729)
0810 22:29:09.297977 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:09.297994 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:09.298088 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:09.298347 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:09.298351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:09.298379 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:09.298525 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.298531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.299488 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.299490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.300753 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.300764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.301688 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.301690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.301825 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.301852 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:29:09.301936 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 22:29:09.302038 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":3,"spinlock_wait_cycles":768}
I20260810 22:29:09.302217 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.297851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:39725 (local address 127.1.221.131:35561)
0810 22:29:09.298282 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:09.298305 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:09.298386 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:09.298799 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:09.298803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:09.298824 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:09.298935 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.298939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.299806 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.299809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.300909 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:09.300917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.301756 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.301758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:09.301853 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.301869 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 22:29:09.301936 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 22:29:09.302038 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":324,"spinlock_wait_cycles":4864}
I20260810 22:29:09.302912 9802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a53a3c25ca41e3aaf0917b15a62e9d" candidate_uuid: "3aa8d37a0d844500aa9fe801dd62efff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4013ce0110a486eb0e2e01623774c2c" is_pre_election: true
I20260810 22:29:09.302922 9951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a53a3c25ca41e3aaf0917b15a62e9d" candidate_uuid: "3aa8d37a0d844500aa9fe801dd62efff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" is_pre_election: true
I20260810 22:29:09.303216 9951 raft_consensus.cc:2454] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa8d37a0d844500aa9fe801dd62efff in term 0.
I20260810 22:29:09.303225 9802 raft_consensus.cc:2454] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa8d37a0d844500aa9fe801dd62efff in term 0.
I20260810 22:29:09.303720 10027 leader_election.cc:302] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa8d37a0d844500aa9fe801dd62efff, a4013ce0110a486eb0e2e01623774c2c; no voters:
I20260810 22:29:09.304032 10198 raft_consensus.cc:2781] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:29:09.304126 10198 raft_consensus.cc:482] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:29:09.304171 10198 raft_consensus.cc:3037] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:09.305250 10198 raft_consensus.cc:504] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } }
I20260810 22:29:09.305590 10198 leader_election.cc:288] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [CANDIDATE]: Term 1 election: Requested vote from peers a4013ce0110a486eb0e2e01623774c2c (127.1.221.129:33893), 9e00f7e5f7ca459a866c8b78fc0df48a (127.1.221.130:39725)
I20260810 22:29:09.305977 9951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a53a3c25ca41e3aaf0917b15a62e9d" candidate_uuid: "3aa8d37a0d844500aa9fe801dd62efff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a"
I20260810 22:29:09.305984 9802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29a53a3c25ca41e3aaf0917b15a62e9d" candidate_uuid: "3aa8d37a0d844500aa9fe801dd62efff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4013ce0110a486eb0e2e01623774c2c"
I20260810 22:29:09.306118 9951 raft_consensus.cc:3037] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:09.306126 9802 raft_consensus.cc:3037] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:09.307200 9951 raft_consensus.cc:2454] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa8d37a0d844500aa9fe801dd62efff in term 1.
I20260810 22:29:09.307202 9802 raft_consensus.cc:2454] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa8d37a0d844500aa9fe801dd62efff in term 1.
I20260810 22:29:09.307510 10028 leader_election.cc:302] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa8d37a0d844500aa9fe801dd62efff, 9e00f7e5f7ca459a866c8b78fc0df48a; no voters:
I20260810 22:29:09.307723 10198 raft_consensus.cc:2781] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:09.307974 10198 raft_consensus.cc:686] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [term 1 LEADER]: Becoming Leader. State: Replica: 3aa8d37a0d844500aa9fe801dd62efff, State: Running, Role: LEADER
I20260810 22:29:09.308959 10198 consensus_queue.cc:237] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } }
I20260810 22:29:09.312664 8792 catalog_manager.cc:5495] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aa8d37a0d844500aa9fe801dd62efff (127.1.221.131). New cstate: current_term: 1 leader_uuid: "3aa8d37a0d844500aa9fe801dd62efff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } health_report { overall_health: UNKNOWN } } }
I20260810 22:29:09.341737 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.338360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.1.221.131:43113)
0810 22:29:09.338504 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.338509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.338523 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:29:09.338603 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.338606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.338662 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.338752 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:29:09.338983 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.339470 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.340275 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.340495 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.340574 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.340856 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:29:09.341300 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:29:09.341397 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.341454 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.341509 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:29:09.347741 9951 raft_consensus.cc:1261] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a [term 1 FOLLOWER]: Refusing update from remote peer 3aa8d37a0d844500aa9fe801dd62efff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:09.347741 9802 raft_consensus.cc:1261] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c [term 1 FOLLOWER]: Refusing update from remote peer 3aa8d37a0d844500aa9fe801dd62efff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:09.348524 10198 consensus_queue.cc:1035] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:09.348726 10199 consensus_queue.cc:1035] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:09.364159 10205 mvcc.cc:205] Tried to move back new op lower bound from 7317098288521093120 to 7317098288372752384. Current Snapshot: MvccSnapshot[applied={T|T < 7317098288521093120 or (T in {7317098288521093120})}]
I20260810 22:29:09.366748 8792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:09.366930 8792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:09.371351 8792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cde8e631773940d9a966d0acdfad3aec: Sending DeleteTablet for 3 replicas of tablet 29a53a3c25ca41e3aaf0917b15a62e9d
I20260810 22:29:09.372192 9780 tablet_service.cc:1514] Processing DeleteTablet for tablet 29a53a3c25ca41e3aaf0917b15a62e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:09 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:34362
I20260810 22:29:09.372545 10075 tablet_service.cc:1514] Processing DeleteTablet for tablet 29a53a3c25ca41e3aaf0917b15a62e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:09 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:45096
I20260810 22:29:09.372822 9928 tablet_service.cc:1514] Processing DeleteTablet for tablet 29a53a3c25ca41e3aaf0917b15a62e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:09 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:44208
I20260810 22:29:09.373446 10215 tablet_replica.cc:330] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a: stopping tablet replica
I20260810 22:29:09.373878 10215 raft_consensus.cc:2229] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:09.374353 10215 raft_consensus.cc:2258] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:09.374766 10214 tablet_replica.cc:330] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff: stopping tablet replica
I20260810 22:29:09.375088 10214 raft_consensus.cc:2229] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:29:09.375277 10213 tablet_replica.cc:330] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c: stopping tablet replica
I20260810 22:29:09.375473 10214 raft_consensus.cc:2258] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:09.375677 10213 raft_consensus.cc:2229] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:09.375736 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 8757
I20260810 22:29:09.379432 10215 ts_tablet_manager.cc:1905] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:09.380014 10213 raft_consensus.cc:2258] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:09.381796 10213 ts_tablet_manager.cc:1905] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:09.382671 10215 ts_tablet_manager.cc:1918] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:09.382795 10215 log.cc:1196] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/29a53a3c25ca41e3aaf0917b15a62e9d
I20260810 22:29:09.383157 10215 ts_tablet_manager.cc:1939] T 29a53a3c25ca41e3aaf0917b15a62e9d P 9e00f7e5f7ca459a866c8b78fc0df48a: Deleting consensus metadata
I20260810 22:29:09.385035 10213 ts_tablet_manager.cc:1918] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:09.385812 10214 ts_tablet_manager.cc:1905] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 22:29:09.388067 9881 connection.cc:535] server connection from 127.0.0.1:44208 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 22:29:09.388257 10213 log.cc:1196] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-0/wal/wals/29a53a3c25ca41e3aaf0917b15a62e9d
I20260810 22:29:09.388743 10213 ts_tablet_manager.cc:1939] T 29a53a3c25ca41e3aaf0917b15a62e9d P a4013ce0110a486eb0e2e01623774c2c: Deleting consensus metadata
I20260810 22:29:09.389649 10214 ts_tablet_manager.cc:1918] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:09.389780 10214 log.cc:1196] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/29a53a3c25ca41e3aaf0917b15a62e9d
I20260810 22:29:09.390134 10214 ts_tablet_manager.cc:1939] T 29a53a3c25ca41e3aaf0917b15a62e9d P 3aa8d37a0d844500aa9fe801dd62efff: Deleting consensus metadata
W20260810 22:29:09.391201 10025 connection.cc:441] server connection from 127.0.0.1:45096 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45096 (request call id 1) could send its response
W20260810 22:29:09.391986 9718 connection.cc:441] server connection from 127.0.0.1:34362 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34362 (request call id 1) could send its response
I20260810 22:29:09.399712 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.373952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.1.221.189:45551)
0810 22:29:09.374070 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.374075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.385480 (+ 11405us) server_negotiation.cc:407] Connection header received
0810 22:29:09.385736 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.385740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.385804 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.385862 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:09.387054 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.387843 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.390619 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.390963 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.391045 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.394975 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:09.394979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:09.394982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:09.395010 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:09.396920 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.397773 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.397777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.397779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.397889 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.398465 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.398468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.398469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.398724 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:09.398868 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.399376 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.399511 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34,"spinlock_wait_cycles":1280}
I20260810 22:29:09.403051 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.393071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.1.221.188:35943)
0810 22:29:09.393185 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.393190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.393692 (+ 502us) server_negotiation.cc:407] Connection header received
0810 22:29:09.393922 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.393928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.393987 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.394044 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:09.394998 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.395844 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.397225 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.397653 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.397738 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.399504 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:09.399507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:09.399510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:09.399533 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:09.401208 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.401726 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.401729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.401731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.401818 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.402109 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.402113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.402115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.402331 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:09.402445 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.402794 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.402901 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:29:09.403434 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.392195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46586 (local address 127.1.221.189:45551)
0810 22:29:09.392330 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.392334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.392528 (+ 194us) server_negotiation.cc:407] Connection header received
0810 22:29:09.392737 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.392740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.392787 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.392836 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:09.394161 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.394996 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.396283 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.396642 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.396715 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.398457 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:09.398461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:09.398463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:09.398498 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:09.401635 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.402109 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.402113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.402115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.402196 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.402537 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.402540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.402542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.402737 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:09.402832 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.403163 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.403291 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":4480}
I20260810 22:29:09.408738 10169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:09.374778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.1.221.188:35943)
0810 22:29:09.374966 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:09.374971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:09.400164 (+ 25193us) server_negotiation.cc:407] Connection header received
0810 22:29:09.400334 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:09.400338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:09.400386 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:09.400437 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:09.401264 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.402057 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.403132 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:09.403494 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:09.403581 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:09.405187 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:09.405190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:09.405192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:09.405216 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:09.406776 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.407247 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.407250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.407252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.407337 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:09.407618 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:09.407621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:09.407623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:09.407787 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:09.407920 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:09.408146 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 22:29:09.408587 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260810 22:29:10.363860 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.363314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.129:50311)
0810 22:29:10.363592 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.363720 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":107,"threads_started":1}
W20260810 22:29:10.364230 9854 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:40671 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
I20260810 22:29:10.365751 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.365227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.131:34379)
0810 22:29:10.365516 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.365637 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
I20260810 22:29:10.366079 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.365620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.130:51009)
0810 22:29:10.365883 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.365982 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
W20260810 22:29:10.366066 10151 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:40671 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
W20260810 22:29:10.366384 10001 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:40671 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
I20260810 22:29:10.872514 10226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cde8e631773940d9a966d0acdfad3aec)
I20260810 22:29:10.872666 10226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } }
I20260810 22:29:10.873162 10226 leader_election.cc:288] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cde8e631773940d9a966d0acdfad3aec (127.1.221.190:40671), 1c7324d97a184a1ba4e86eb31802017e (127.1.221.189:45551)
I20260810 22:29:10.873904 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.873397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:36026)
0810 22:29:10.873663 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.873783 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
W20260810 22:29:10.874233 9642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
I20260810 22:29:10.874377 10230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cde8e631773940d9a966d0acdfad3aec)
I20260810 22:29:10.874615 10230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } }
I20260810 22:29:10.875219 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.874968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:36032)
0810 22:29:10.875091 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.875153 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 22:29:10.875422 9642 leader_election.cc:334] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cde8e631773940d9a966d0acdfad3aec (127.1.221.190:40671): Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
I20260810 22:29:10.876124 10230 leader_election.cc:288] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cde8e631773940d9a966d0acdfad3aec (127.1.221.190:40671), 0da636ffd07b4335b5158a89e987d75e (127.1.221.188:35943)
I20260810 22:29:10.876382 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.875939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:36040)
0810 22:29:10.876218 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.876294 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":74,"threads_started":1}
W20260810 22:29:10.876547 9437 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
I20260810 22:29:10.877422 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.877167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:36054)
0810 22:29:10.877301 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.877358 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 22:29:10.877525 9437 leader_election.cc:334] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cde8e631773940d9a966d0acdfad3aec (127.1.221.190:40671): Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
I20260810 22:29:10.878556 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.873949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.1.221.189:45551)
0810 22:29:10.874282 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:10.874288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:10.874307 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:29:10.874368 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:10.874371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:10.874422 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:10.874903 (+ 481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:10.875030 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.875725 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.877343 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.877956 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.878083 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.878143 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:10.878367 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 22:29:10.878443 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":75,"threads_started":1}
I20260810 22:29:10.878566 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.873793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:45551 (local address 127.0.0.1:46602)
0810 22:29:10.874013 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.874029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:10.874153 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:10.874605 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:10.874611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:10.874634 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:10.874775 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.874785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.875843 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.875849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.877155 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.877172 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.878099 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.878103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.878236 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.878255 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:29:10.878348 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 22:29:10.878449 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":24,"thread_start_us":77,"threads_started":1}
I20260810 22:29:10.879124 9468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0da636ffd07b4335b5158a89e987d75e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1c7324d97a184a1ba4e86eb31802017e" is_pre_election: true
I20260810 22:29:10.879273 9468 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da636ffd07b4335b5158a89e987d75e in term 1.
I20260810 22:29:10.879587 9642 leader_election.cc:302] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da636ffd07b4335b5158a89e987d75e, 1c7324d97a184a1ba4e86eb31802017e; no voters: cde8e631773940d9a966d0acdfad3aec
I20260810 22:29:10.879806 10226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 22:29:10.879874 10226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 FOLLOWER]: Starting leader election (detected failure of leader cde8e631773940d9a966d0acdfad3aec)
I20260810 22:29:10.879932 10226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:29:10.880980 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.875980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35943 (local address 127.0.0.1:58618)
0810 22:29:10.876373 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.876384 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:10.876491 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:10.877463 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:10.877466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:10.877484 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:10.877619 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.877625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.878521 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.878526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.879838 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.879855 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.880675 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.880677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.880743 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.880763 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 22:29:10.880829 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 22:29:10.880880 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":49,"threads_started":1}
I20260810 22:29:10.881065 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.876984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.1.221.188:35943)
0810 22:29:10.877222 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:10.877232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:10.877246 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:29:10.877302 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:10.877305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:10.877350 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:10.877451 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:10.877763 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.878410 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.879979 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.880566 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.880668 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.880721 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:10.880857 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 22:29:10.880926 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":172,"thread_start_us":100,"threads_started":1}
I20260810 22:29:10.881450 10226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } }
I20260810 22:29:10.881597 9682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1c7324d97a184a1ba4e86eb31802017e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0da636ffd07b4335b5158a89e987d75e" is_pre_election: true
I20260810 22:29:10.881819 9682 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c7324d97a184a1ba4e86eb31802017e in current term 2: Already voted for candidate 0da636ffd07b4335b5158a89e987d75e in this term.
I20260810 22:29:10.881865 10226 leader_election.cc:288] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [CANDIDATE]: Term 2 election: Requested vote from peers cde8e631773940d9a966d0acdfad3aec (127.1.221.190:40671), 1c7324d97a184a1ba4e86eb31802017e (127.1.221.189:45551)
I20260810 22:29:10.882261 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.882003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:36066)
0810 22:29:10.882122 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.882175 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 22:29:10.882222 9438 leader_election.cc:302] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c7324d97a184a1ba4e86eb31802017e; no voters: 0da636ffd07b4335b5158a89e987d75e, cde8e631773940d9a966d0acdfad3aec
I20260810 22:29:10.882439 10230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:29:10.882805 9468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0da636ffd07b4335b5158a89e987d75e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1c7324d97a184a1ba4e86eb31802017e"
I20260810 22:29:10.882954 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.882682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:36074)
0810 22:29:10.882849 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.882903 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260810 22:29:10.883098 9642 leader_election.cc:334] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cde8e631773940d9a966d0acdfad3aec (127.1.221.190:40671): Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
I20260810 22:29:10.883504 10230 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:29:10.884439 9468 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da636ffd07b4335b5158a89e987d75e in term 2.
I20260810 22:29:10.884734 9642 leader_election.cc:302] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0da636ffd07b4335b5158a89e987d75e, 1c7324d97a184a1ba4e86eb31802017e; no voters: cde8e631773940d9a966d0acdfad3aec
I20260810 22:29:10.884917 10226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:29:10.885135 10226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [term 2 LEADER]: Becoming Leader. State: Replica: 0da636ffd07b4335b5158a89e987d75e, State: Running, Role: LEADER
I20260810 22:29:10.885396 10226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } }
I20260810 22:29:10.886740 10237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0da636ffd07b4335b5158a89e987d75e. Latest consensus state: current_term: 2 leader_uuid: "0da636ffd07b4335b5158a89e987d75e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } } }
I20260810 22:29:10.886868 10237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:10.887197 10239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:10.889564 10239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:10.890051 10239 catalog_manager.cc:1259] Loaded cluster ID: a1199cc48bca4f2c86716b1fe896bbf4
I20260810 22:29:10.890100 10239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:10.890676 10239 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:10.891088 10239 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: Loaded TSK: 0
I20260810 22:29:10.891950 10239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:29:10.917199 9669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:29:10.920701 9669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:29:10.924046 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.923743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:36098)
0810 22:29:10.923895 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.923963 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 22:29:10.924046 9468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [term 2 FOLLOWER]: Refusing update from remote peer 0da636ffd07b4335b5158a89e987d75e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 22:29:10.924612 10237 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 22:29:10.924991 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.924723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.0.0.1:36112)
0810 22:29:10.924866 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.924932 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 22:29:10.925161 9642 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e -> Peer cde8e631773940d9a966d0acdfad3aec (127.1.221.190:40671): Couldn't send request to peer cde8e631773940d9a966d0acdfad3aec. Status: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:29:10.926301 10230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0da636ffd07b4335b5158a89e987d75e. Latest consensus state: current_term: 2 leader_uuid: "0da636ffd07b4335b5158a89e987d75e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } } }
I20260810 22:29:10.926455 10230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:29:10.927361 10240 mvcc.cc:205] Tried to move back new op lower bound from 7317098294979162112 to 7317098294829383680. Current Snapshot: MvccSnapshot[applied={T|T < 7317098294979162112}]
I20260810 22:29:10.927629 10226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0da636ffd07b4335b5158a89e987d75e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } } }
I20260810 22:29:10.927762 10226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:10.928151 10230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0da636ffd07b4335b5158a89e987d75e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cde8e631773940d9a966d0acdfad3aec" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40671 } } peers { permanent_uuid: "1c7324d97a184a1ba4e86eb31802017e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 45551 } attrs { promote: false } } peers { permanent_uuid: "0da636ffd07b4335b5158a89e987d75e" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35943 } attrs { promote: false } } }
I20260810 22:29:10.928280 10230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c7324d97a184a1ba4e86eb31802017e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:29:10.928261 10242 mvcc.cc:205] Tried to move back new op lower bound from 7317098294979162112 to 7317098294829383680. Current Snapshot: MvccSnapshot[applied={T|T < 7317098294979162112}]
I20260810 22:29:10.938308 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.933999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:39725 (local address 127.0.0.1:44216)
0810 22:29:10.934273 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.934289 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:10.934373 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:10.934945 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:10.934949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:10.934966 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:10.935091 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.935098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.936090 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.936093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.936896 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.936905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.937901 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.937905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.938002 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.938028 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:29:10.938073 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 22:29:10.938142 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":3}
I20260810 22:29:10.938876 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.934289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.1.221.130:39725)
0810 22:29:10.934619 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:10.934624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:10.934639 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:29:10.934696 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:10.934699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:10.934757 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:10.935204 (+ 447us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:10.935340 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.935966 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.937085 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.937767 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.938630 (+ 863us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.938684 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:10.938714 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 22:29:10.938773 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":92,"threads_started":1}
I20260810 22:29:10.939435 9928 tablet_service.cc:1467] Processing CreateTablet for tablet a9d803f03f8d441eaf004c1dd1bac6ae (DEFAULT_TABLE table=test-table [id=a06583d1d8c4415eb68357bb686ef357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:10.939523 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.934659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:43113 (local address 127.0.0.1:45116)
0810 22:29:10.934967 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.934980 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:10.935046 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:10.936001 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:10.936007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:10.936032 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:10.936136 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.936143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.937087 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.937091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.938200 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.938210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.939098 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.939102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.939216 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.939260 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 22:29:10.939310 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 22:29:10.939387 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":5}
I20260810 22:29:10.939786 9928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d803f03f8d441eaf004c1dd1bac6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:10.939832 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.934807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.1.221.129:33893)
0810 22:29:10.935339 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:10.935343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:10.935355 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:29:10.935400 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:10.935402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:10.935439 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:10.935545 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:10.936104 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.936881 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.938388 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.939004 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.939106 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.939170 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:10.939582 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 22:29:10.939687 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":97,"threads_started":1}
I20260810 22:29:10.940054 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.934682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33893 (local address 127.0.0.1:34370)
0810 22:29:10.935025 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.935039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:10.935126 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:10.935553 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:10.935556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:10.935572 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:10.935680 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.935686 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.937007 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.937013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.938281 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.938291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.939224 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.939228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.939351 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.939370 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:29:10.939431 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 22:29:10.939491 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":78,"thread_start_us":107,"threads_started":1}
I20260810 22:29:10.940121 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.935063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.1.221.131:43113)
0810 22:29:10.935775 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:10.935779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:10.935792 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:29:10.935846 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:10.935848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:10.935887 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:10.935953 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:10.936263 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.936996 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.938329 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.938947 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.939795 (+ 848us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.939863 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:10.939902 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 22:29:10.939992 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":73,"threads_started":1}
I20260810 22:29:10.940647 10075 tablet_service.cc:1467] Processing CreateTablet for tablet a9d803f03f8d441eaf004c1dd1bac6ae (DEFAULT_TABLE table=test-table [id=a06583d1d8c4415eb68357bb686ef357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:10.940824 9780 tablet_service.cc:1467] Processing CreateTablet for tablet a9d803f03f8d441eaf004c1dd1bac6ae (DEFAULT_TABLE table=test-table [id=a06583d1d8c4415eb68357bb686ef357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:10.940920 10075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d803f03f8d441eaf004c1dd1bac6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:10.941126 9780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d803f03f8d441eaf004c1dd1bac6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:10.942811 10251 tablet_bootstrap.cc:492] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a: Bootstrap starting.
I20260810 22:29:10.943626 10252 tablet_bootstrap.cc:492] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff: Bootstrap starting.
I20260810 22:29:10.943902 10251 tablet_bootstrap.cc:654] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:10.943975 10253 tablet_bootstrap.cc:492] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c: Bootstrap starting.
I20260810 22:29:10.944844 10253 tablet_bootstrap.cc:654] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:10.944952 10252 tablet_bootstrap.cc:654] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:10.946139 10251 tablet_bootstrap.cc:492] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a: No bootstrap required, opened a new log
I20260810 22:29:10.946139 10253 tablet_bootstrap.cc:492] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c: No bootstrap required, opened a new log
I20260810 22:29:10.946233 10253 ts_tablet_manager.cc:1397] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:29:10.946317 10251 ts_tablet_manager.cc:1397] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 22:29:10.946756 10253 raft_consensus.cc:348] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } }
I20260810 22:29:10.946756 10251 raft_consensus.cc:348] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } }
I20260810 22:29:10.946888 10253 raft_consensus.cc:374] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:10.946934 10253 raft_consensus.cc:729] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4013ce0110a486eb0e2e01623774c2c, State: Initialized, Role: FOLLOWER
I20260810 22:29:10.946943 10251 raft_consensus.cc:374] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:10.947017 10251 raft_consensus.cc:729] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e00f7e5f7ca459a866c8b78fc0df48a, State: Initialized, Role: FOLLOWER
I20260810 22:29:10.947085 10253 consensus_queue.cc:260] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } }
I20260810 22:29:10.947216 10251 consensus_queue.cc:260] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } }
I20260810 22:29:10.947445 10253 ts_tablet_manager.cc:1428] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 22:29:10.948208 10251 ts_tablet_manager.cc:1428] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 22:29:10.950318 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.948114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.129:40415)
0810 22:29:10.950124 (+ 2010us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.950186 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1932,"thread_start_us":96,"threads_started":1}
I20260810 22:29:10.950899 9853 heartbeater.cc:503] Master 127.1.221.188:35943 was elected leader, sending a full tablet report...
I20260810 22:29:10.951510 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.949448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.130:40549)
0810 22:29:10.951340 (+ 1892us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.951409 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1822,"thread_start_us":94,"threads_started":1}
I20260810 22:29:10.951812 10000 heartbeater.cc:503] Master 127.1.221.188:35943 was elected leader, sending a full tablet report...
I20260810 22:29:10.952297 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.951984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.130:58693)
0810 22:29:10.952139 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.952199 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 22:29:10.953358 10252 tablet_bootstrap.cc:492] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff: No bootstrap required, opened a new log
I20260810 22:29:10.953477 10252 ts_tablet_manager.cc:1397] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260810 22:29:10.953950 10252 raft_consensus.cc:348] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } }
I20260810 22:29:10.954078 10252 raft_consensus.cc:374] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:10.954124 10252 raft_consensus.cc:729] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa8d37a0d844500aa9fe801dd62efff, State: Initialized, Role: FOLLOWER
I20260810 22:29:10.954267 10252 consensus_queue.cc:260] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } }
I20260810 22:29:10.954646 10252 ts_tablet_manager.cc:1428] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 22:29:10.955420 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.954982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.131:34325)
0810 22:29:10.955255 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.955319 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":127,"threads_started":1}
I20260810 22:29:10.956156 10149 heartbeater.cc:503] Master 127.1.221.188:35943 was elected leader, sending a full tablet report...
I20260810 22:29:10.956449 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.956164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.131:36443)
0810 22:29:10.956311 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.956368 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 22:29:10.958739 10259 raft_consensus.cc:482] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:29:10.958868 10259 raft_consensus.cc:504] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } }
I20260810 22:29:10.960410 10259 leader_election.cc:288] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e00f7e5f7ca459a866c8b78fc0df48a (127.1.221.130:39725), 3aa8d37a0d844500aa9fe801dd62efff (127.1.221.131:43113)
I20260810 22:29:10.963310 10260 raft_consensus.cc:482] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:29:10.963426 10260 raft_consensus.cc:504] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } }
I20260810 22:29:10.964619 10260 leader_election.cc:288] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4013ce0110a486eb0e2e01623774c2c (127.1.221.129:33893), 3aa8d37a0d844500aa9fe801dd62efff (127.1.221.131:43113)
I20260810 22:29:10.964711 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.961034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:45189 (local address 127.1.221.131:43113)
0810 22:29:10.961145 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:10.961152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:10.961180 (+ 28us) server_negotiation.cc:407] Connection header received
0810 22:29:10.961227 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:10.961230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:10.961269 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:10.961372 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:10.961703 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.962202 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.963386 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.963998 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.964116 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.964171 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:10.964485 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 22:29:10.964559 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:29:10.964761 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.960652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:44713 (local address 127.1.221.130:39725)
0810 22:29:10.960768 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:10.960772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:10.961092 (+ 320us) server_negotiation.cc:407] Connection header received
0810 22:29:10.961216 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:10.961219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:10.961269 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:10.961328 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:10.961672 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.962195 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.963517 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.964072 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.964161 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.964207 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:10.964552 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 22:29:10.964632 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 22:29:10.965189 10265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.960550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:39725 (local address 127.1.221.129:44713)
0810 22:29:10.960976 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.960992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:10.961119 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:10.961370 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:10.961373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:10.961390 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:10.961519 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.961525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.962310 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.962316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.963170 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.963180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.964308 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.964311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.964408 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.964430 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:29:10.964930 (+ 500us) client_negotiation.cc:240] Negotiation successful
0810 22:29:10.965026 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"spinlock_wait_cycles":2816}
I20260810 22:29:10.965519 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.960644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:43113 (local address 127.1.221.129:45189)
0810 22:29:10.960997 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.961009 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:10.961129 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:10.961399 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:10.961402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:10.961420 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:10.961532 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.961537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.962342 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.962345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.963267 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.963292 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.964259 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.964262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.964343 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.964362 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:29:10.964457 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 22:29:10.965359 (+ 902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":51,"spinlock_wait_cycles":3072,"thread_start_us":117,"threads_started":1}
I20260810 22:29:10.965929 9951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d803f03f8d441eaf004c1dd1bac6ae" candidate_uuid: "a4013ce0110a486eb0e2e01623774c2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" is_pre_election: true
I20260810 22:29:10.966078 9951 raft_consensus.cc:2454] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4013ce0110a486eb0e2e01623774c2c in term 0.
I20260810 22:29:10.966140 10097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d803f03f8d441eaf004c1dd1bac6ae" candidate_uuid: "a4013ce0110a486eb0e2e01623774c2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa8d37a0d844500aa9fe801dd62efff" is_pre_election: true
I20260810 22:29:10.966380 10097 raft_consensus.cc:2454] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4013ce0110a486eb0e2e01623774c2c in term 0.
I20260810 22:29:10.966503 9720 leader_election.cc:302] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e00f7e5f7ca459a866c8b78fc0df48a, a4013ce0110a486eb0e2e01623774c2c; no voters:
I20260810 22:29:10.966774 10259 raft_consensus.cc:2781] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:29:10.966830 10259 raft_consensus.cc:482] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:29:10.966864 10259 raft_consensus.cc:3037] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:10.968202 10259 raft_consensus.cc:504] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } }
I20260810 22:29:10.968842 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.964974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:34457 (local address 127.1.221.129:33893)
0810 22:29:10.965081 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:10.965085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:10.965097 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:29:10.965164 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:10.965168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:10.965211 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:10.965283 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:10.965608 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.966281 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.967719 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.968282 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.968413 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.968458 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:10.968667 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 22:29:10.968738 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 22:29:10.968887 10259 leader_election.cc:288] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [CANDIDATE]: Term 1 election: Requested vote from peers 9e00f7e5f7ca459a866c8b78fc0df48a (127.1.221.130:39725), 3aa8d37a0d844500aa9fe801dd62efff (127.1.221.131:43113)
I20260810 22:29:10.968856 9951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d803f03f8d441eaf004c1dd1bac6ae" candidate_uuid: "a4013ce0110a486eb0e2e01623774c2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a"
I20260810 22:29:10.968986 9951 raft_consensus.cc:3037] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:10.968869 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.964742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33893 (local address 127.1.221.130:34457)
0810 22:29:10.964890 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.964906 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:10.964992 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:10.965334 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:10.965337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:10.965358 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:10.965450 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.965456 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.966375 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.966378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.967575 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.967587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.968410 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.968413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.968518 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.968541 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:29:10.968648 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 22:29:10.968735 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 22:29:10.969249 10097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d803f03f8d441eaf004c1dd1bac6ae" candidate_uuid: "a4013ce0110a486eb0e2e01623774c2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa8d37a0d844500aa9fe801dd62efff"
I20260810 22:29:10.969367 10097 raft_consensus.cc:3037] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:10.969480 9802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d803f03f8d441eaf004c1dd1bac6ae" candidate_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4013ce0110a486eb0e2e01623774c2c" is_pre_election: true
I20260810 22:29:10.969704 9802 raft_consensus.cc:2379] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e00f7e5f7ca459a866c8b78fc0df48a in current term 1: Already voted for candidate a4013ce0110a486eb0e2e01623774c2c in this term.
I20260810 22:29:10.970218 9951 raft_consensus.cc:2454] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4013ce0110a486eb0e2e01623774c2c in term 1.
I20260810 22:29:10.970611 10097 raft_consensus.cc:2454] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4013ce0110a486eb0e2e01623774c2c in term 1.
I20260810 22:29:10.970911 9718 leader_election.cc:302] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa8d37a0d844500aa9fe801dd62efff, a4013ce0110a486eb0e2e01623774c2c; no voters:
I20260810 22:29:10.971087 10259 raft_consensus.cc:2781] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:10.971362 10259 raft_consensus.cc:686] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [term 1 LEADER]: Becoming Leader. State: Replica: a4013ce0110a486eb0e2e01623774c2c, State: Running, Role: LEADER
I20260810 22:29:10.971793 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.965237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:46497 (local address 127.1.221.131:43113)
0810 22:29:10.965667 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:10.965673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:10.966788 (+ 1115us) server_negotiation.cc:407] Connection header received
0810 22:29:10.966974 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:10.966977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:10.967017 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:10.967116 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:10.967392 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.968022 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.970618 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.971233 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.971346 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.971393 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:10.971617 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 22:29:10.971686 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316}
I20260810 22:29:10.972114 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.964890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:43113 (local address 127.1.221.130:46497)
0810 22:29:10.966709 (+ 1819us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.966723 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:10.966848 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:10.967129 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:10.967132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:10.967148 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:10.967277 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.967282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.968160 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.968164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.970491 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:10.970502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.971343 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.971347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:10.971463 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.971491 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 22:29:10.971903 (+ 412us) client_negotiation.cc:240] Negotiation successful
0810 22:29:10.971984 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"thread_start_us":109,"threads_started":1}
I20260810 22:29:10.972190 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.971838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.129:36565)
0810 22:29:10.972071 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.972125 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260810 22:29:10.972157 10259 consensus_queue.cc:237] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } } peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } }
I20260810 22:29:10.972491 10097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d803f03f8d441eaf004c1dd1bac6ae" candidate_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aa8d37a0d844500aa9fe801dd62efff" is_pre_election: true
I20260810 22:29:10.972664 10097 raft_consensus.cc:2379] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e00f7e5f7ca459a866c8b78fc0df48a in current term 1: Already voted for candidate a4013ce0110a486eb0e2e01623774c2c in this term.
I20260810 22:29:10.973125 9879 leader_election.cc:302] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e00f7e5f7ca459a866c8b78fc0df48a; no voters: 3aa8d37a0d844500aa9fe801dd62efff, a4013ce0110a486eb0e2e01623774c2c
I20260810 22:29:10.973624 10260 raft_consensus.cc:2726] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:29:10.975419 9669 catalog_manager.cc:5495] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c reported cstate change: term changed from 0 to 1, leader changed from <none> to a4013ce0110a486eb0e2e01623774c2c (127.1.221.129). New cstate: current_term: 1 leader_uuid: "a4013ce0110a486eb0e2e01623774c2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4013ce0110a486eb0e2e01623774c2c" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 } health_report { overall_health: UNKNOWN } } }
W20260810 22:29:10.989682 9856 tablet.cc:2369] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:29:10.991321 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.987441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.1.221.129:33893)
0810 22:29:10.987569 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:10.987572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:10.987587 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:29:10.987734 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:10.987737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:10.987801 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:10.987896 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:29:10.988231 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.988717 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.989702 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:10.990074 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:10.990165 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:10.990493 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:29:10.990968 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:29:10.991058 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:10.991121 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 22:29:10.991166 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:29:10.994330 9951 raft_consensus.cc:1261] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [term 1 FOLLOWER]: Refusing update from remote peer a4013ce0110a486eb0e2e01623774c2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:10.994501 10097 raft_consensus.cc:1261] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff [term 1 FOLLOWER]: Refusing update from remote peer a4013ce0110a486eb0e2e01623774c2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:10.994946 10259 consensus_queue.cc:1035] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e00f7e5f7ca459a866c8b78fc0df48a" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 39725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:10.995479 10259 consensus_queue.cc:1035] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa8d37a0d844500aa9fe801dd62efff" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:10.997073 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.996828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.130:50971)
0810 22:29:10.996917 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.996980 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 22:29:10.999459 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:10.998604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.131:56419)
0810 22:29:10.999316 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:10.999378 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":628}
I20260810 22:29:10.999688 10270 mvcc.cc:205] Tried to move back new op lower bound from 7317098295267233792 to 7317098295187316736. Current Snapshot: MvccSnapshot[applied={T|T < 7317098295267233792}]
I20260810 22:29:11.000311 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:11.000051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.129:46117)
0810 22:29:11.000194 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:11.000246 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 22:29:11.002728 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:11.001831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.130:49993)
0810 22:29:11.002597 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:11.002657 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":690}
I20260810 22:29:11.003031 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:11.002693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.129:40921)
0810 22:29:11.002902 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:11.002955 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260810 22:29:11.004288 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:11.003937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40671 (local address 127.1.221.131:58833)
0810 22:29:11.004154 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:11.004207 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260810 22:29:11.011456 9671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:11.011631 9671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:11.015919 9671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0da636ffd07b4335b5158a89e987d75e: Sending DeleteTablet for 3 replicas of tablet a9d803f03f8d441eaf004c1dd1bac6ae
I20260810 22:29:11.016633 9928 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d803f03f8d441eaf004c1dd1bac6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:11 UTC) from {username='oryx', principal='oryx/127.1.221.188@KRBTEST.COM'} at 127.0.0.1:44216
I20260810 22:29:11.016937 10075 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d803f03f8d441eaf004c1dd1bac6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:11 UTC) from {username='oryx', principal='oryx/127.1.221.188@KRBTEST.COM'} at 127.0.0.1:45116
I20260810 22:29:11.017007 9780 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d803f03f8d441eaf004c1dd1bac6ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:11 UTC) from {username='oryx', principal='oryx/127.1.221.188@KRBTEST.COM'} at 127.0.0.1:34370
I20260810 22:29:11.017237 10280 tablet_replica.cc:330] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a: stopping tablet replica
I20260810 22:29:11.017385 10282 tablet_replica.cc:330] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c: stopping tablet replica
I20260810 22:29:11.017391 10280 raft_consensus.cc:2229] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:11.017547 10280 raft_consensus.cc:2258] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:11.017607 10282 raft_consensus.cc:2229] T a9d803f03f8d441eaf004c1dd1bac6ae P a4013ce0110a486eb0e2e01623774c2c [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:29:11.017863 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 9614
I20260810 22:29:11.019635 10280 ts_tablet_manager.cc:1905] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:11.022451 10281 tablet_replica.cc:330] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff: stopping tablet replica
I20260810 22:29:11.022611 10281 raft_consensus.cc:2229] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:11.022763 10281 raft_consensus.cc:2258] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:11.022970 10280 ts_tablet_manager.cc:1918] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:11.023061 10280 log.cc:1196] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/a9d803f03f8d441eaf004c1dd1bac6ae
I20260810 22:29:11.023360 10280 ts_tablet_manager.cc:1939] T a9d803f03f8d441eaf004c1dd1bac6ae P 9e00f7e5f7ca459a866c8b78fc0df48a: Deleting consensus metadata
I20260810 22:29:11.023547 10281 ts_tablet_manager.cc:1905] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:11.024099 9644 catalog_manager.cc:4841] TS 9e00f7e5f7ca459a866c8b78fc0df48a (127.1.221.130:39725): tablet a9d803f03f8d441eaf004c1dd1bac6ae (table test-table [id=a06583d1d8c4415eb68357bb686ef357]) successfully deleted
I20260810 22:29:11.025720 10281 ts_tablet_manager.cc:1918] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:11.025847 10281 log.cc:1196] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/a9d803f03f8d441eaf004c1dd1bac6ae
I20260810 22:29:11.026121 10281 ts_tablet_manager.cc:1939] T a9d803f03f8d441eaf004c1dd1bac6ae P 3aa8d37a0d844500aa9fe801dd62efff: Deleting consensus metadata
I20260810 22:29:11.026760 9642 catalog_manager.cc:4841] TS 3aa8d37a0d844500aa9fe801dd62efff (127.1.221.131:43113): tablet a9d803f03f8d441eaf004c1dd1bac6ae (table test-table [id=a06583d1d8c4415eb68357bb686ef357]) successfully deleted
I20260810 22:29:11.030138 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 9862
I20260810 22:29:11.031373 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:11.030251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33893 (local address 127.0.0.1:34390)
0810 22:29:11.031207 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:11.031264 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":866}
W20260810 22:29:11.032146 9643 catalog_manager.cc:4584] TS a4013ce0110a486eb0e2e01623774c2c (127.1.221.129:33893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9d803f03f8d441eaf004c1dd1bac6ae: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33893: connect: Connection refused (error 111)
I20260810 22:29:11.039347 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 10007
I20260810 22:29:11.048995 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 9301
I20260810 22:29:11.056636 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 9424
2026-08-10T22:29:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5434 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:29:11 dist-test-slave-njxd krb5kdc[10285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:29:11 dist-test-slave-njxd krb5kdc[10285](info): set up 2 sockets
Aug 10 22:29:11 dist-test-slave-njxd krb5kdc[10285](info): commencing operation
krb5kdc: starting...
W20260810 22:29:12.159613 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:29:12 dist-test-slave-njxd krb5kdc[10285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:29:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.190.keytab.
Entry for principal oryx/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.190.keytab.
I20260810 22:29:12.285459 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39405
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.190.keytab
--principal=oryx/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:39405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:12.400394 10302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:12.400663 10302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:12.400703 10302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:12.404739 10302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:12.404801 10302 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 22:29:12.404820 10302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:12.404837 10302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:12.404902 10302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:12.404930 10302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:29:12.410229 10302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:39405
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.190.keytab
--principal=oryx/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:12.411293 10302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:12.412348 10302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:12.420982 10308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:12.420982 10307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:12.421005 10310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:12.421175 10302 server_base.cc:1034] running on GCE node
I20260810 22:29:12.422003 10302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:12.423079 10302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:12.424290 10302 hybrid_clock.cc:648] HybridClock initialized: now 1786400952424266 us; error 52 us; skew 500 ppm
Aug 10 22:29:12 dist-test-slave-njxd krb5kdc[10285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:12.430297 10302 init.cc:377] Logged in from keytab as oryx/127.1.221.190@KRBTEST.COM (short username oryx)
I20260810 22:29:12.431260 10302 webserver.cc:460] Webserver started at http://127.1.221.190:43515/ using document root <none> and password file <none>
I20260810 22:29:12.431890 10302 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:12.431989 10302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:12.432300 10302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:12.434522 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "3736b230c488481b8dfeb8f9e17acd6e"
format_stamp: "Formatted at 2026-08-10 22:29:12 on dist-test-slave-njxd"
server_key: "bbd25c7731bf5fa0e26e63d78e7fee40"
server_key_iv: "7433a81b5e13d6fb15567feba6f32a1f"
server_key_version: "encryptionkey@0"
I20260810 22:29:12.435127 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "3736b230c488481b8dfeb8f9e17acd6e"
format_stamp: "Formatted at 2026-08-10 22:29:12 on dist-test-slave-njxd"
server_key: "bbd25c7731bf5fa0e26e63d78e7fee40"
server_key_iv: "7433a81b5e13d6fb15567feba6f32a1f"
server_key_version: "encryptionkey@0"
I20260810 22:29:12.439313 10302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:29:12.441869 10319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:12.443176 10302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:29:12.443356 10302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "3736b230c488481b8dfeb8f9e17acd6e"
format_stamp: "Formatted at 2026-08-10 22:29:12 on dist-test-slave-njxd"
server_key: "bbd25c7731bf5fa0e26e63d78e7fee40"
server_key_iv: "7433a81b5e13d6fb15567feba6f32a1f"
server_key_version: "encryptionkey@0"
I20260810 22:29:12.443478 10302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:12.459980 10302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:12.463372 10302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:12.463587 10302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:12.472641 10302 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:39405
I20260810 22:29:12.472666 10385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:39405 every 8 connection(s)
I20260810 22:29:12.473657 10302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:29:12.476435 10386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:12.482188 10386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e: Bootstrap starting.
I20260810 22:29:12.482331 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 10302
I20260810 22:29:12.482457 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:29:12.482775 1910 external_mini_cluster.cc:1428] Setting key 91f8765d1b95758ac84449fda455c46a
I20260810 22:29:12.485028 10386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:12.485858 10386 log.cc:824] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:12.488509 10386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e: No bootstrap required, opened a new log
Aug 10 22:29:12 dist-test-slave-njxd krb5kdc[10285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260810 22:29:12.492806 10386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3736b230c488481b8dfeb8f9e17acd6e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39405 } }
I20260810 22:29:12.493130 10386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:12.493217 10386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3736b230c488481b8dfeb8f9e17acd6e, State: Initialized, Role: FOLLOWER
I20260810 22:29:12.493765 10386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3736b230c488481b8dfeb8f9e17acd6e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39405 } }
I20260810 22:29:12.493945 10386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:29:12.494050 10386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:29:12.494153 10386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:12.495157 10386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3736b230c488481b8dfeb8f9e17acd6e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39405 } }
I20260810 22:29:12.495553 10386 leader_election.cc:302] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3736b230c488481b8dfeb8f9e17acd6e; no voters:
I20260810 22:29:12.495895 10386 leader_election.cc:288] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:29:12.496016 10391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:12.496316 10391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [term 1 LEADER]: Becoming Leader. State: Replica: 3736b230c488481b8dfeb8f9e17acd6e, State: Running, Role: LEADER
I20260810 22:29:12.496629 10391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3736b230c488481b8dfeb8f9e17acd6e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39405 } }
I20260810 22:29:12.497174 10386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:12.497893 10391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3736b230c488481b8dfeb8f9e17acd6e. Latest consensus state: current_term: 1 leader_uuid: "3736b230c488481b8dfeb8f9e17acd6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3736b230c488481b8dfeb8f9e17acd6e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39405 } } }
I20260810 22:29:12.498028 10391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:12.497918 10392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3736b230c488481b8dfeb8f9e17acd6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3736b230c488481b8dfeb8f9e17acd6e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39405 } } }
I20260810 22:29:12.498102 10392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:12.498401 10399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:12.499132 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:12.484165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.1.221.190:39405)
0810 22:29:12.484651 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:12.484668 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:12.484703 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:29:12.485415 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:12.485435 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:12.485842 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:12.486196 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:12.487082 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:12.488539 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:12.489430 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:12.490260 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:12.490449 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:12.493266 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:12.493273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:12.493291 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:12.493323 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:12.495206 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:12.495709 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:12.495715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:12.495722 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:12.495897 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:12.496266 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:12.496270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:12.496272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:12.496804 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:12.496962 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:12.497465 (+ 503us) server_negotiation.cc:299] Negotiation successful
0810 22:29:12.497816 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":147,"threads_started":1}
I20260810 22:29:12.501628 10399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:12.507138 10399 catalog_manager.cc:1347] Generated new cluster ID: 542a13c3191e4ff4a872deb9e416cd1d
I20260810 22:29:12.507221 10399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:12.523358 10399 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:29:12.524339 10399 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:12.539502 10399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e: Generated new TSK 0
I20260810 22:29:12.540253 10399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab.
Entry for principal oryx/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab.
I20260810 22:29:12.577407 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39405
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:12.713979 10413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:12.714279 10413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:12.714365 10413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:12.718380 10413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:12.718467 10413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:12.718537 10413 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:29:12.718626 10413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:29:12.723811 10413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:39405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:12.724856 10413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:12.725893 10413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:12.735533 10418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:12.735582 10413 server_base.cc:1034] running on GCE node
W20260810 22:29:12.735563 10421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:12.735534 10419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:12.736228 10413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:12.736881 10413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:12.738075 10413 hybrid_clock.cc:648] HybridClock initialized: now 1786400952738062 us; error 25 us; skew 500 ppm
Aug 10 22:29:12 dist-test-slave-njxd krb5kdc[10285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:12.740890 10413 init.cc:377] Logged in from keytab as oryx/127.1.221.129@KRBTEST.COM (short username oryx)
I20260810 22:29:12.741852 10413 webserver.cc:460] Webserver started at http://127.1.221.129:41077/ using document root <none> and password file <none>
I20260810 22:29:12.742425 10413 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:12.742524 10413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:12.742769 10413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:12.744841 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "d3e09cde00ff4994a737d17b384928d3"
format_stamp: "Formatted at 2026-08-10 22:29:12 on dist-test-slave-njxd"
server_key: "2fd4f62446f8f26558007e24beec6e5e"
server_key_iv: "4f917e6cf3f3b1a9a17964528a39e854"
server_key_version: "encryptionkey@0"
I20260810 22:29:12.745424 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "d3e09cde00ff4994a737d17b384928d3"
format_stamp: "Formatted at 2026-08-10 22:29:12 on dist-test-slave-njxd"
server_key: "2fd4f62446f8f26558007e24beec6e5e"
server_key_iv: "4f917e6cf3f3b1a9a17964528a39e854"
server_key_version: "encryptionkey@0"
I20260810 22:29:12.749125 10413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:29:12.751499 10428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:12.752697 10413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:29:12.752846 10413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "d3e09cde00ff4994a737d17b384928d3"
format_stamp: "Formatted at 2026-08-10 22:29:12 on dist-test-slave-njxd"
server_key: "2fd4f62446f8f26558007e24beec6e5e"
server_key_iv: "4f917e6cf3f3b1a9a17964528a39e854"
server_key_version: "encryptionkey@0"
I20260810 22:29:12.752960 10413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:12.771783 10413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:12.774940 10413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:12.775125 10413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:12.776535 10413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:12.776585 10413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:12.776625 10413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:12.776684 10413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:29:12 dist-test-slave-njxd krb5kdc[10285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260810 22:29:12.788226 10413 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:36321
I20260810 22:29:12.788321 10552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:36321 every 8 connection(s)
I20260810 22:29:12.789359 10413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:29:12.791198 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:12.777420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:59567 (local address 127.1.221.190:39405)
0810 22:29:12.777551 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:12.777555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:12.778644 (+ 1089us) server_negotiation.cc:407] Connection header received
0810 22:29:12.779768 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:12.779772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:12.779836 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:12.779904 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:12.781708 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:12.782302 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:12.783552 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:12.783910 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:12.784058 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:12.787202 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:12.787206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:12.787209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:12.787236 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:12.789040 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:12.789621 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:12.789624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:12.789626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:12.789731 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:12.790141 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:12.790144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:12.790146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:12.790389 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:12.790475 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:12.790972 (+ 497us) server_negotiation.cc:299] Negotiation successful
0810 22:29:12.791077 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:29:12.791957 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:12.778006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39405 (local address 127.1.221.129:59567)
0810 22:29:12.778498 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:12.778539 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:12.779515 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:12.780393 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:12.780403 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:12.780850 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:12.781539 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:12.781564 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:12.782442 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:12.782446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:12.783426 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:12.783435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:12.784094 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:12.784098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:12.784288 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:12.784921 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:12.784946 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:12.786994 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:12.789165 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:12.789169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:12.789181 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:12.789483 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:12.789843 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:12.789846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:12.789848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:12.790016 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:12.790511 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:12.790517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:12.790748 (+ 231us) client_negotiation.cc:769] Sending connection context
0810 22:29:12.790906 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 22:29:12.791147 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":143,"threads_started":1}
I20260810 22:29:12.793184 10554 heartbeater.cc:347] Connected to a master server at 127.1.221.190:39405
I20260810 22:29:12.793329 10554 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:12.793915 10554 heartbeater.cc:511] Master 127.1.221.190:39405 requested a full tablet report, sending...
I20260810 22:29:12.795598 10336 ts_manager.cc:194] Registered new tserver with Master: d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321)
I20260810 22:29:12.795642 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 10413
I20260810 22:29:12.795776 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:29:12.796125 1910 external_mini_cluster.cc:1428] Setting key 05fedc0e6cd2d84f722a540e94c64474
I20260810 22:29:12.797158 10336 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:59567
I20260810 22:29:12.803830 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:12.795632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:49001 (local address 127.1.221.190:39405)
0810 22:29:12.795749 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:12.795752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:12.795986 (+ 234us) server_negotiation.cc:407] Connection header received
0810 22:29:12.796105 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:12.796108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:12.796148 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:12.796222 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:12.796925 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:12.797643 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:12.798816 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:12.799122 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:12.799200 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:12.800686 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:12.800689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:12.800691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:12.800713 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:12.802112 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:12.802651 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:12.802653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:12.802654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:12.802728 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:12.803049 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:12.803051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:12.803053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:12.803207 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:12.803282 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:12.803622 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 22:29:12.803713 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:29:12.804077 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:12.795590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39405 (local address 127.1.221.129:49001)
0810 22:29:12.795899 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:12.795911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:12.796027 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:12.796343 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:12.796346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:12.796618 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:12.796759 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:12.796780 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:12.797757 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:12.797760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:12.798654 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:12.798665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:12.799357 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:12.799361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:12.799458 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:12.800042 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:12.800062 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:12.800538 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:12.802235 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:12.802242 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:12.802247 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:12.802537 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:12.802829 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:12.802835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:12.802840 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:12.802951 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:12.803320 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:12.803326 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:12.803446 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 22:29:12.803571 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 22:29:12.803730 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for oryx/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab.
Entry for principal oryx/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab.
I20260810 22:29:12.825804 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39405
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:12.939607 10563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:12.939930 10563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:12.940011 10563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:12.944180 10563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:12.944279 10563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:12.944355 10563 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:29:12.944514 10563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:29:12.951107 10563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:39405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:12.952244 10563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:12.953325 10563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:12.963346 10568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:12.963364 10563 server_base.cc:1034] running on GCE node
W20260810 22:29:12.963364 10571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:12.963366 10569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:12.963994 10563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:12.964658 10563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:12.965852 10563 hybrid_clock.cc:648] HybridClock initialized: now 1786400952965838 us; error 25 us; skew 500 ppm
Aug 10 22:29:12 dist-test-slave-njxd krb5kdc[10285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:12.968587 10563 init.cc:377] Logged in from keytab as oryx/127.1.221.130@KRBTEST.COM (short username oryx)
I20260810 22:29:12.969434 10563 webserver.cc:460] Webserver started at http://127.1.221.130:36563/ using document root <none> and password file <none>
I20260810 22:29:12.970042 10563 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:12.970108 10563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:12.970360 10563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:12.972431 10563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "f0a2c92f89b24aea8d135775e09c8c7e"
format_stamp: "Formatted at 2026-08-10 22:29:12 on dist-test-slave-njxd"
server_key: "fd49e042285ea1e89638984ff0c9ba54"
server_key_iv: "fd41c9922492b1295a7e7d16de9c91d4"
server_key_version: "encryptionkey@0"
I20260810 22:29:12.973026 10563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "f0a2c92f89b24aea8d135775e09c8c7e"
format_stamp: "Formatted at 2026-08-10 22:29:12 on dist-test-slave-njxd"
server_key: "fd49e042285ea1e89638984ff0c9ba54"
server_key_iv: "fd41c9922492b1295a7e7d16de9c91d4"
server_key_version: "encryptionkey@0"
I20260810 22:29:12.976856 10563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:29:12.979414 10578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:12.980647 10563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:29:12.980804 10563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "f0a2c92f89b24aea8d135775e09c8c7e"
format_stamp: "Formatted at 2026-08-10 22:29:12 on dist-test-slave-njxd"
server_key: "fd49e042285ea1e89638984ff0c9ba54"
server_key_iv: "fd41c9922492b1295a7e7d16de9c91d4"
server_key_version: "encryptionkey@0"
I20260810 22:29:12.980927 10563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:12.997308 10563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:13.000970 10563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:13.001190 10563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:13.003170 10563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:13.003240 10563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:13.003314 10563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:13.003346 10563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:13.013628 10563 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:36553
I20260810 22:29:13.013727 10703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:36553 every 8 connection(s)
Aug 10 22:29:13 dist-test-slave-njxd krb5kdc[10285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260810 22:29:13.014819 10563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:29:13.018635 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.003704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:36189 (local address 127.1.221.190:39405)
0810 22:29:13.003893 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.003899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.005027 (+ 1128us) server_negotiation.cc:407] Connection header received
0810 22:29:13.006253 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.006259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.006332 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.006435 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:13.008368 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.009299 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.010953 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.011348 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.011482 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.014715 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:13.014718 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:13.014720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:13.014747 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:13.016548 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:13.017037 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:13.017040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:13.017041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:13.017122 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:13.017431 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:13.017433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:13.017434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:13.017682 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:13.017805 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.018326 (+ 521us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.018459 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 22:29:13.019393 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.004360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39405 (local address 127.1.221.130:36189)
0810 22:29:13.004844 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:13.004895 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:13.005999 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:13.006781 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:13.006793 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:13.007283 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:13.008154 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.008181 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.009468 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.009473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.010807 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.010819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.011528 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.011533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.011783 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.012495 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:13.012515 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:13.014481 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:13.016658 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:13.016663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:13.016675 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:13.016937 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:13.017206 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:13.017208 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:13.017209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:13.017343 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:13.017808 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:13.017817 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:13.018063 (+ 246us) client_negotiation.cc:769] Sending connection context
0810 22:29:13.018254 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 22:29:13.018582 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":140,"threads_started":1}
I20260810 22:29:13.020566 10704 heartbeater.cc:347] Connected to a master server at 127.1.221.190:39405
I20260810 22:29:13.020735 10704 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:13.021306 10704 heartbeater.cc:511] Master 127.1.221.190:39405 requested a full tablet report, sending...
I20260810 22:29:13.022433 10337 ts_manager.cc:194] Registered new tserver with Master: f0a2c92f89b24aea8d135775e09c8c7e (127.1.221.130:36553)
I20260810 22:29:13.023195 10337 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:36189
I20260810 22:29:13.023741 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 10563
I20260810 22:29:13.023878 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:29:13.024227 1910 external_mini_cluster.cc:1428] Setting key d763ca6802748bc2bc12b265dae3907e
I20260810 22:29:13.030833 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.022942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:39635 (local address 127.1.221.190:39405)
0810 22:29:13.023070 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.023074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.023336 (+ 262us) server_negotiation.cc:407] Connection header received
0810 22:29:13.023464 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.023468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.023526 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.023599 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:13.024206 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.024860 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.025908 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.026125 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.026209 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.027533 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:13.027536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:13.027538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:13.027560 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:13.029048 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:13.029506 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:13.029510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:13.029511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:13.029651 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:13.029955 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:13.029958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:13.029960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:13.030178 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:13.030289 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.030564 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.030674 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:29:13.030845 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.022808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39405 (local address 127.1.221.130:39635)
0810 22:29:13.023232 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:13.023247 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:13.023350 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:13.023708 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:13.023710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:13.023964 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:13.024081 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.024087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.024999 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.025003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.025767 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.025778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.026259 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.026265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.026392 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.026910 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:13.026930 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:13.027389 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:13.029171 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:13.029175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:13.029177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:13.029393 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:13.029752 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:13.029755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:13.029757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:13.029849 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:13.030286 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:13.030289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:13.030392 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 22:29:13.030506 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 22:29:13.030662 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for oryx/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab.
Entry for principal oryx/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab.
I20260810 22:29:13.053247 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39405
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:42221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:13.171250 10714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:13.171528 10714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:13.171573 10714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:13.175714 10714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:13.175772 10714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:13.175812 10714 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:29:13.175845 10714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:29:13.181098 10714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.221.190:39405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:13.182298 10714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:13.183382 10714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:13.193154 10722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:13.193166 10720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:13.193166 10719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:13.193445 10714 server_base.cc:1034] running on GCE node
I20260810 22:29:13.193878 10714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:13.194538 10714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:13.195744 10714 hybrid_clock.cc:648] HybridClock initialized: now 1786400953195734 us; error 32 us; skew 500 ppm
Aug 10 22:29:13 dist-test-slave-njxd krb5kdc[10285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:13.198907 10714 init.cc:377] Logged in from keytab as oryx/127.1.221.131@KRBTEST.COM (short username oryx)
I20260810 22:29:13.199873 10714 webserver.cc:460] Webserver started at http://127.1.221.131:40951/ using document root <none> and password file <none>
I20260810 22:29:13.200466 10714 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:13.200529 10714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:13.200831 10714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:13.202983 10714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "cdd4645b07b84e4c875c9886f306159c"
format_stamp: "Formatted at 2026-08-10 22:29:13 on dist-test-slave-njxd"
server_key: "9b361cf4fa3974581d94031e60513721"
server_key_iv: "6ddf4ad03d3522639f31ccc9067cfb73"
server_key_version: "encryptionkey@0"
I20260810 22:29:13.203569 10714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "cdd4645b07b84e4c875c9886f306159c"
format_stamp: "Formatted at 2026-08-10 22:29:13 on dist-test-slave-njxd"
server_key: "9b361cf4fa3974581d94031e60513721"
server_key_iv: "6ddf4ad03d3522639f31ccc9067cfb73"
server_key_version: "encryptionkey@0"
I20260810 22:29:13.207345 10714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:29:13.209920 10729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:13.211167 10714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:29:13.211308 10714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "cdd4645b07b84e4c875c9886f306159c"
format_stamp: "Formatted at 2026-08-10 22:29:13 on dist-test-slave-njxd"
server_key: "9b361cf4fa3974581d94031e60513721"
server_key_iv: "6ddf4ad03d3522639f31ccc9067cfb73"
server_key_version: "encryptionkey@0"
I20260810 22:29:13.211424 10714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:13.226963 10714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:13.229991 10714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:13.230201 10714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:13.231724 10714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:13.231798 10714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:13.231873 10714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:13.231928 10714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:29:13 dist-test-slave-njxd krb5kdc[10285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400953, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260810 22:29:13.244097 10714 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:32833
I20260810 22:29:13.244179 10860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:32833 every 8 connection(s)
I20260810 22:29:13.245241 10714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:29:13.246628 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.232460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:49565 (local address 127.1.221.190:39405)
0810 22:29:13.232593 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.232597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.233780 (+ 1183us) server_negotiation.cc:407] Connection header received
0810 22:29:13.234913 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.234917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.234981 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.235051 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:13.236471 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.237106 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.239175 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.239413 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.239535 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.242525 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:13.242529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:13.242537 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:13.242563 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:13.244398 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:13.244932 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:13.244936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:13.244938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:13.245031 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:13.245415 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:13.245418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:13.245420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:13.245674 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:13.245799 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.246401 (+ 602us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.246508 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:29:13.247524 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.233050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39405 (local address 127.1.221.131:49565)
0810 22:29:13.233556 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:13.233670 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:13.234634 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:13.235437 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:13.235444 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:13.235764 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:13.236315 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.236328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.237256 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.237264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.239026 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.239037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.239591 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.239596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.239832 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.240503 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:13.240529 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:13.242348 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:13.244516 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:13.244520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:13.244531 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:13.244789 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:13.245140 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:13.245142 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:13.245143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:13.245313 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:13.245854 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:13.245860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:13.246140 (+ 280us) client_negotiation.cc:769] Sending connection context
0810 22:29:13.246320 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 22:29:13.246675 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":156,"threads_started":1}
I20260810 22:29:13.248739 10861 heartbeater.cc:347] Connected to a master server at 127.1.221.190:39405
I20260810 22:29:13.248911 10861 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:13.249406 10861 heartbeater.cc:511] Master 127.1.221.190:39405 requested a full tablet report, sending...
I20260810 22:29:13.250479 10336 ts_manager.cc:194] Registered new tserver with Master: cdd4645b07b84e4c875c9886f306159c (127.1.221.131:32833)
I20260810 22:29:13.250996 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 10714
I20260810 22:29:13.251128 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:29:13.251287 10336 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:49565
I20260810 22:29:13.251482 1910 external_mini_cluster.cc:1428] Setting key b11c36ded0135e7237be29344a7b1d0b
I20260810 22:29:13.253566 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:29:13.258874 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.251041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:55839 (local address 127.1.221.190:39405)
0810 22:29:13.251171 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.251177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.251425 (+ 248us) server_negotiation.cc:407] Connection header received
0810 22:29:13.251598 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.251601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.251658 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.251743 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:13.252313 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.252935 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.253992 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.254225 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.254347 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.255453 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:13.255457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:13.255459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:13.255487 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:13.257179 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:13.257707 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:13.257710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:13.257711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:13.257789 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:13.258102 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:13.258104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:13.258105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:13.258261 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:13.258346 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.258666 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.258760 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 22:29:13.259063 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.251000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39405 (local address 127.1.221.131:55839)
0810 22:29:13.251327 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:13.251340 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:13.251492 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:13.251760 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:13.251763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:13.252014 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:13.252146 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.252153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.253077 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.253079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.253863 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.253873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.254349 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.254352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.254445 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.255003 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:13.255014 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:13.255314 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:13.257317 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:13.257320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:13.257322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:13.257616 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:13.257891 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:13.257897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:13.257901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:13.258010 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:13.258370 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:13.258375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:13.258492 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 22:29:13.258609 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 22:29:13.258780 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":153,"threads_started":1}
Aug 10 22:29:13 dist-test-slave-njxd krb5kdc[10285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400953, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:29:13 dist-test-slave-njxd krb5kdc[10285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:13 dist-test-slave-njxd krb5kdc[10285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 22:29:13.280364 10389 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:13.274105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.1.221.190:39405)
0810 22:29:13.274237 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.274241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.274252 (+ 11us) server_negotiation.cc:407] Connection header received
0810 22:29:13.274298 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.274300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.274339 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.274402 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:13.275143 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.275661 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.276487 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.276703 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.276778 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.280158 (+ 3380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37490: BlockingRecv error: recv got EOF from 127.0.0.1:37490 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 22:29:13 dist-test-slave-njxd krb5kdc[10285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400953, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260810 22:29:13.289769 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.281801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.1.221.190:39405)
0810 22:29:13.281958 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.281964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.282106 (+ 142us) server_negotiation.cc:407] Connection header received
0810 22:29:13.282262 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.282265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.282322 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.282406 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:13.283016 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.283543 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.284369 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.284592 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.284719 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.287103 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:13.287106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:13.287108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:13.287132 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:13.288311 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:13.288700 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:13.288703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:13.288704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:13.288780 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:13.289019 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:13.289021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:13.289022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:13.289185 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:13.289284 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.289488 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.289647 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:29:13.292784 10336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:29:13.296101 10336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:29:13.311591 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.305096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:32833 (local address 127.0.0.1:33102)
0810 22:29:13.306371 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:13.306387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:13.306549 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:13.307157 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:13.307160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:13.307177 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:13.307288 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.307294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.308694 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.308725 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.310055 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.310066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.311050 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.311052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.311142 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.311172 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 22:29:13.311322 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 22:29:13.311442 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":126,"thread_start_us":90,"threads_started":1}
I20260810 22:29:13.311602 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.305319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33102 (local address 127.1.221.131:32833)
0810 22:29:13.305709 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.305725 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.306474 (+ 749us) server_negotiation.cc:407] Connection header received
0810 22:29:13.306684 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.306689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.306873 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.306982 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:13.307410 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.308556 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.310191 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.310928 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.311045 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.311208 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.311360 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.311456 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20260810 22:29:13.312032 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.305759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36553 (local address 127.0.0.1:54012)
0810 22:29:13.307188 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:13.307201 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:13.307291 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:13.307743 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:13.307745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:13.307763 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:13.307850 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.307857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.308851 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.308854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.310368 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.310380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.311631 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.311635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.311723 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.311747 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:29:13.311828 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 22:29:13.311896 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1370,"mutex_wait_us":138,"thread_start_us":120,"threads_started":1}
I20260810 22:29:13.312523 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.305985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.1.221.130:36553)
0810 22:29:13.306627 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.306642 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.307392 (+ 750us) server_negotiation.cc:407] Connection header received
0810 22:29:13.307473 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.307477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.307613 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.307764 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:13.307945 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.308721 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.310502 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.311318 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.311417 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.311566 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.312269 (+ 703us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.312369 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":132,"threads_started":1}
I20260810 22:29:13.312676 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.304998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36321 (local address 127.0.0.1:46602)
0810 22:29:13.305827 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:13.305881 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:13.306006 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:13.306973 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:13.306980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:13.307005 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:13.307148 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.307159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.308669 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.308672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.310368 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.310380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.312301 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.312305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.312393 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.312417 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:29:13.312468 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 22:29:13.312536 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":142,"thread_start_us":98,"threads_started":1}
I20260810 22:29:13.313277 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.306021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46602 (local address 127.1.221.129:36321)
0810 22:29:13.306471 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.306478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.306500 (+ 22us) server_negotiation.cc:407] Connection header received
0810 22:29:13.306582 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.306590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.306805 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.306967 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:13.307309 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.308538 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.311302 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.312153 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.312861 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.312996 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.313042 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.313129 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":83,"threads_started":1}
I20260810 22:29:13.315263 10787 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6b12a27b68447095b8a84541611d72 (DEFAULT_TABLE table=test-table [id=03a9dee0433d4bfa9daae56a6d08ce14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:13.315270 10631 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6b12a27b68447095b8a84541611d72 (DEFAULT_TABLE table=test-table [id=03a9dee0433d4bfa9daae56a6d08ce14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:13.315804 10484 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6b12a27b68447095b8a84541611d72 (DEFAULT_TABLE table=test-table [id=03a9dee0433d4bfa9daae56a6d08ce14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:13.316380 10631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6b12a27b68447095b8a84541611d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:13.316394 10787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6b12a27b68447095b8a84541611d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:13.316709 10484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6b12a27b68447095b8a84541611d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:13.323292 10889 tablet_bootstrap.cc:492] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c: Bootstrap starting.
I20260810 22:29:13.324529 10891 tablet_bootstrap.cc:492] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e: Bootstrap starting.
I20260810 22:29:13.325763 10889 tablet_bootstrap.cc:654] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:13.325922 10890 tablet_bootstrap.cc:492] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3: Bootstrap starting.
I20260810 22:29:13.326475 10891 tablet_bootstrap.cc:654] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:13.326694 10889 log.cc:824] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:13.327329 10891 log.cc:824] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:13.327983 10890 tablet_bootstrap.cc:654] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:13.328662 10890 log.cc:824] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:13.328896 10889 tablet_bootstrap.cc:492] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c: No bootstrap required, opened a new log
I20260810 22:29:13.329097 10889 ts_tablet_manager.cc:1397] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260810 22:29:13.329300 10891 tablet_bootstrap.cc:492] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e: No bootstrap required, opened a new log
I20260810 22:29:13.329479 10891 ts_tablet_manager.cc:1397] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 22:29:13.330438 10890 tablet_bootstrap.cc:492] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3: No bootstrap required, opened a new log
I20260810 22:29:13.330632 10890 ts_tablet_manager.cc:1397] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 22:29:13.333189 10891 raft_consensus.cc:348] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } }
I20260810 22:29:13.333154 10889 raft_consensus.cc:348] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } }
I20260810 22:29:13.333432 10889 raft_consensus.cc:374] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:13.333432 10891 raft_consensus.cc:374] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:13.333485 10889 raft_consensus.cc:729] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4645b07b84e4c875c9886f306159c, State: Initialized, Role: FOLLOWER
I20260810 22:29:13.333495 10891 raft_consensus.cc:729] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a2c92f89b24aea8d135775e09c8c7e, State: Initialized, Role: FOLLOWER
I20260810 22:29:13.333966 10891 consensus_queue.cc:260] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } }
I20260810 22:29:13.333962 10889 consensus_queue.cc:260] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } }
I20260810 22:29:13.334389 10890 raft_consensus.cc:348] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } }
I20260810 22:29:13.334682 10890 raft_consensus.cc:374] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:13.334757 10890 raft_consensus.cc:729] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e09cde00ff4994a737d17b384928d3, State: Initialized, Role: FOLLOWER
I20260810 22:29:13.335078 10891 ts_tablet_manager.cc:1428] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:29:13.335170 10704 heartbeater.cc:503] Master 127.1.221.190:39405 was elected leader, sending a full tablet report...
I20260810 22:29:13.335179 10890 consensus_queue.cc:260] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } }
I20260810 22:29:13.335865 10554 heartbeater.cc:503] Master 127.1.221.190:39405 was elected leader, sending a full tablet report...
I20260810 22:29:13.335011 10889 ts_tablet_manager.cc:1428] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:29:13.336158 10890 ts_tablet_manager.cc:1428] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 22:29:13.335327 10861 heartbeater.cc:503] Master 127.1.221.190:39405 was elected leader, sending a full tablet report...
W20260810 22:29:13.496312 10862 tablet.cc:2369] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:29:13.516132 10705 tablet.cc:2369] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:29:13.540915 10555 tablet.cc:2369] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:29:13.621099 10896 raft_consensus.cc:482] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:29:13.621402 10896 raft_consensus.cc:504] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } }
I20260810 22:29:13.622713 10896 leader_election.cc:288] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321), cdd4645b07b84e4c875c9886f306159c (127.1.221.131:32833)
I20260810 22:29:13.626811 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.623082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:50311 (local address 127.1.221.129:36321)
0810 22:29:13.623251 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.623255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.623430 (+ 175us) server_negotiation.cc:407] Connection header received
0810 22:29:13.623607 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.623611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.623661 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.623752 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:13.624040 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.624559 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.625763 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.626167 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.626312 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.626368 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.626608 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.626682 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 22:29:13.626809 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.623148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:49931 (local address 127.1.221.131:32833)
0810 22:29:13.623297 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.623302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.623540 (+ 238us) server_negotiation.cc:407] Connection header received
0810 22:29:13.623696 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.623699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.623752 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.623845 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:13.624102 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.624616 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.625766 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.626178 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.626322 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.626372 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.626602 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.626674 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:29:13.626845 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.622944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36321 (local address 127.1.221.130:50311)
0810 22:29:13.623330 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:13.623344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:13.623492 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:13.623770 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:13.623773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:13.623798 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:13.623925 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.623931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.624688 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.624691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.625632 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.625643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.626300 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.626303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.626399 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.626498 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 22:29:13.626589 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 22:29:13.626665 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":89,"thread_start_us":127,"threads_started":1}
I20260810 22:29:13.626848 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.623006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:32833 (local address 127.1.221.130:49931)
0810 22:29:13.623448 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:13.623461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:13.623583 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:13.623858 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:13.623861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:13.623877 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:13.623992 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.623998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.624738 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.624742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.625638 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.625648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.626317 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.626320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.626439 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.626493 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 22:29:13.626583 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 22:29:13.626660 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":55,"thread_start_us":75,"threads_started":1}
I20260810 22:29:13.627503 10505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b12a27b68447095b8a84541611d72" candidate_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e09cde00ff4994a737d17b384928d3" is_pre_election: true
I20260810 22:29:13.627698 10811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b12a27b68447095b8a84541611d72" candidate_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4645b07b84e4c875c9886f306159c" is_pre_election: true
I20260810 22:29:13.627837 10505 raft_consensus.cc:2454] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a2c92f89b24aea8d135775e09c8c7e in term 0.
I20260810 22:29:13.628031 10811 raft_consensus.cc:2454] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a2c92f89b24aea8d135775e09c8c7e in term 0.
I20260810 22:29:13.628384 10580 leader_election.cc:302] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3e09cde00ff4994a737d17b384928d3, f0a2c92f89b24aea8d135775e09c8c7e; no voters:
I20260810 22:29:13.628677 10896 raft_consensus.cc:2781] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:29:13.628764 10896 raft_consensus.cc:482] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:29:13.628809 10896 raft_consensus.cc:3037] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:13.629964 10896 raft_consensus.cc:504] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } }
I20260810 22:29:13.630324 10896 leader_election.cc:288] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [CANDIDATE]: Term 1 election: Requested vote from peers d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321), cdd4645b07b84e4c875c9886f306159c (127.1.221.131:32833)
I20260810 22:29:13.630681 10811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b12a27b68447095b8a84541611d72" candidate_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4645b07b84e4c875c9886f306159c"
I20260810 22:29:13.630676 10505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6b12a27b68447095b8a84541611d72" candidate_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e09cde00ff4994a737d17b384928d3"
I20260810 22:29:13.630807 10811 raft_consensus.cc:3037] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:13.630820 10505 raft_consensus.cc:3037] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:13.632216 10505 raft_consensus.cc:2454] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a2c92f89b24aea8d135775e09c8c7e in term 1.
I20260810 22:29:13.632568 10580 leader_election.cc:302] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3e09cde00ff4994a737d17b384928d3, f0a2c92f89b24aea8d135775e09c8c7e; no voters:
I20260810 22:29:13.632735 10811 raft_consensus.cc:2454] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a2c92f89b24aea8d135775e09c8c7e in term 1.
I20260810 22:29:13.632859 10896 raft_consensus.cc:2781] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:13.633108 10896 raft_consensus.cc:686] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [term 1 LEADER]: Becoming Leader. State: Replica: f0a2c92f89b24aea8d135775e09c8c7e, State: Running, Role: LEADER
I20260810 22:29:13.633448 10896 consensus_queue.cc:237] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } }
I20260810 22:29:13.636795 10337 catalog_manager.cc:5495] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e reported cstate change: term changed from 0 to 1, leader changed from <none> to f0a2c92f89b24aea8d135775e09c8c7e (127.1.221.130). New cstate: current_term: 1 leader_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } health_report { overall_health: HEALTHY } } }
I20260810 22:29:13.674207 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.670867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.1.221.130:36553)
0810 22:29:13.671017 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.671021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.671035 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:29:13.671117 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.671119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.671178 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.671269 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:29:13.671512 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.672026 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.672841 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.673049 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.673125 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.673381 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:29:13.673861 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:29:13.673949 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.674044 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.674098 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:29:13.680281 10505 raft_consensus.cc:1261] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Refusing update from remote peer f0a2c92f89b24aea8d135775e09c8c7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:13.680262 10811 raft_consensus.cc:1261] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Refusing update from remote peer f0a2c92f89b24aea8d135775e09c8c7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:13.681042 10896 consensus_queue.cc:1035] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:13.681298 10900 consensus_queue.cc:1035] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:13.687975 10903 mvcc.cc:205] Tried to move back new op lower bound from 7317098306267013120 to 7317098306085158912. Current Snapshot: MvccSnapshot[applied={T|T < 7317098306267013120}]
I20260810 22:29:13.689759 10904 mvcc.cc:205] Tried to move back new op lower bound from 7317098306267013120 to 7317098306085158912. Current Snapshot: MvccSnapshot[applied={T|T < 7317098306267013120}]
I20260810 22:29:13.697052 10337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:13.697238 10337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:13.700023 10337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e: Sending DeleteTablet for 3 replicas of tablet 3b6b12a27b68447095b8a84541611d72
I20260810 22:29:13.700845 10484 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6b12a27b68447095b8a84541611d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:13 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:46602
I20260810 22:29:13.701071 10631 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6b12a27b68447095b8a84541611d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:13 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:54012
I20260810 22:29:13.701385 10787 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6b12a27b68447095b8a84541611d72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:13 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:33102
I20260810 22:29:13.701540 10912 tablet_replica.cc:330] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3: stopping tablet replica
I20260810 22:29:13.701911 10914 tablet_replica.cc:330] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c: stopping tablet replica
I20260810 22:29:13.701898 10912 raft_consensus.cc:2229] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:13.701923 10337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:29:13.702262 10912 raft_consensus.cc:2258] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 22:29:13.702354 10337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:29:13.702543 10914 raft_consensus.cc:2229] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:13.702922 10914 raft_consensus.cc:2258] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:13.703915 10912 ts_tablet_manager.cc:1905] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:13.704527 10914 ts_tablet_manager.cc:1905] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:13.707396 10913 tablet_replica.cc:330] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e: stopping tablet replica
I20260810 22:29:13.707500 10631 tablet_service.cc:1467] Processing CreateTablet for tablet b9fe773d9cfe424aa80a0a8e4aeac163 (DEFAULT_TABLE table=test-table [id=737623b13fe9489ea4039d0c2cf3c4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:13.707517 10484 tablet_service.cc:1467] Processing CreateTablet for tablet b9fe773d9cfe424aa80a0a8e4aeac163 (DEFAULT_TABLE table=test-table [id=737623b13fe9489ea4039d0c2cf3c4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:13.707614 10787 tablet_service.cc:1467] Processing CreateTablet for tablet b9fe773d9cfe424aa80a0a8e4aeac163 (DEFAULT_TABLE table=test-table [id=737623b13fe9489ea4039d0c2cf3c4f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:13.707687 10913 raft_consensus.cc:2229] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:29:13.707795 10484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fe773d9cfe424aa80a0a8e4aeac163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:13.707798 10631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fe773d9cfe424aa80a0a8e4aeac163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:13.707849 10787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fe773d9cfe424aa80a0a8e4aeac163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:13.708104 10913 raft_consensus.cc:2258] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:13.709703 10890 tablet_bootstrap.cc:492] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3: Bootstrap starting.
I20260810 22:29:13.710009 10912 ts_tablet_manager.cc:1918] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:13.710194 10891 tablet_bootstrap.cc:492] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e: Bootstrap starting.
I20260810 22:29:13.710384 10889 tablet_bootstrap.cc:492] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c: Bootstrap starting.
I20260810 22:29:13.710734 10890 tablet_bootstrap.cc:654] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:13.711185 10891 tablet_bootstrap.cc:654] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:13.711354 10889 tablet_bootstrap.cc:654] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:13.712193 10912 log.cc:1196] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/wal/wals/3b6b12a27b68447095b8a84541611d72
I20260810 22:29:13.712239 10890 tablet_bootstrap.cc:492] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3: No bootstrap required, opened a new log
I20260810 22:29:13.712313 10890 ts_tablet_manager.cc:1397] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:29:13.712580 10912 ts_tablet_manager.cc:1939] T 3b6b12a27b68447095b8a84541611d72 P d3e09cde00ff4994a737d17b384928d3: Deleting consensus metadata
I20260810 22:29:13.713083 10890 raft_consensus.cc:348] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.713198 10890 raft_consensus.cc:374] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:13.713231 10890 raft_consensus.cc:729] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e09cde00ff4994a737d17b384928d3, State: Initialized, Role: FOLLOWER
I20260810 22:29:13.713353 10890 consensus_queue.cc:260] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.713624 10890 ts_tablet_manager.cc:1428] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 22:29:13.714052 10914 ts_tablet_manager.cc:1918] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:13.714212 10914 log.cc:1196] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/3b6b12a27b68447095b8a84541611d72
I20260810 22:29:13.714486 10889 tablet_bootstrap.cc:492] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c: No bootstrap required, opened a new log
I20260810 22:29:13.714637 10889 ts_tablet_manager.cc:1397] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260810 22:29:13.715106 10889 raft_consensus.cc:348] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.715227 10889 raft_consensus.cc:374] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:13.715265 10889 raft_consensus.cc:729] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4645b07b84e4c875c9886f306159c, State: Initialized, Role: FOLLOWER
I20260810 22:29:13.715420 10889 consensus_queue.cc:260] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.715448 10321 catalog_manager.cc:4841] TS d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321): tablet 3b6b12a27b68447095b8a84541611d72 (table test-table [id=03a9dee0433d4bfa9daae56a6d08ce14]) successfully deleted
I20260810 22:29:13.715679 10889 ts_tablet_manager.cc:1428] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:29:13.716473 10891 tablet_bootstrap.cc:492] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e: No bootstrap required, opened a new log
I20260810 22:29:13.716557 10891 ts_tablet_manager.cc:1397] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 22:29:13.716864 10913 ts_tablet_manager.cc:1905] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:13.716986 10891 raft_consensus.cc:348] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.717089 10891 raft_consensus.cc:374] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:13.717125 10891 raft_consensus.cc:729] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a2c92f89b24aea8d135775e09c8c7e, State: Initialized, Role: FOLLOWER
I20260810 22:29:13.717247 10891 consensus_queue.cc:260] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.717522 10891 ts_tablet_manager.cc:1428] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:29:13.719520 10914 ts_tablet_manager.cc:1939] T 3b6b12a27b68447095b8a84541611d72 P cdd4645b07b84e4c875c9886f306159c: Deleting consensus metadata
I20260810 22:29:13.720289 10913 ts_tablet_manager.cc:1918] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:13.720393 10913 log.cc:1196] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/3b6b12a27b68447095b8a84541611d72
I20260810 22:29:13.720757 10913 ts_tablet_manager.cc:1939] T 3b6b12a27b68447095b8a84541611d72 P f0a2c92f89b24aea8d135775e09c8c7e: Deleting consensus metadata
I20260810 22:29:13.720813 10322 catalog_manager.cc:4841] TS cdd4645b07b84e4c875c9886f306159c (127.1.221.131:32833): tablet 3b6b12a27b68447095b8a84541611d72 (table test-table [id=03a9dee0433d4bfa9daae56a6d08ce14]) successfully deleted
I20260810 22:29:13.721690 10320 catalog_manager.cc:4841] TS f0a2c92f89b24aea8d135775e09c8c7e (127.1.221.130:36553): tablet 3b6b12a27b68447095b8a84541611d72 (table test-table [id=03a9dee0433d4bfa9daae56a6d08ce14]) successfully deleted
I20260810 22:29:13.738557 10895 raft_consensus.cc:482] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:29:13.738705 10895 raft_consensus.cc:504] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.740199 10895 leader_election.cc:288] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321), f0a2c92f89b24aea8d135775e09c8c7e (127.1.221.130:36553)
I20260810 22:29:13.745019 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.740375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36321 (local address 127.1.221.131:60037)
0810 22:29:13.740814 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:13.740831 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:13.740940 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:13.741197 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:13.741200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:13.741221 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:13.741373 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.741380 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.742444 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.742447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.743823 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.743836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.744563 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.744567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.744683 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.744713 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 22:29:13.744784 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 22:29:13.744868 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":92}
I20260810 22:29:13.745097 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.740437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36553 (local address 127.1.221.131:37941)
0810 22:29:13.740873 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:13.740887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:13.741009 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:13.741363 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:13.741366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:13.741380 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:13.741500 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.741506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.742266 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.742271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.743824 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.743835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.744712 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.744715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.744822 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.744841 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:29:13.744907 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 22:29:13.744966 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":105,"thread_start_us":112,"threads_started":1}
I20260810 22:29:13.745165 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.740679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:37941 (local address 127.1.221.130:36553)
0810 22:29:13.740823 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.740828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.740947 (+ 119us) server_negotiation.cc:407] Connection header received
0810 22:29:13.741131 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.741134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.741186 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.741236 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:13.741644 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.742142 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.743962 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.744591 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.744710 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.744762 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.744949 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.745028 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:29:13.745811 10655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe773d9cfe424aa80a0a8e4aeac163" candidate_uuid: "cdd4645b07b84e4c875c9886f306159c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" is_pre_election: true
I20260810 22:29:13.746065 10655 raft_consensus.cc:2454] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd4645b07b84e4c875c9886f306159c in term 0.
I20260810 22:29:13.746189 10887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.740531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:60037 (local address 127.1.221.129:36321)
0810 22:29:13.740671 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.740676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.740898 (+ 222us) server_negotiation.cc:407] Connection header received
0810 22:29:13.741039 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.741043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.741105 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.741184 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:13.741506 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.742328 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.744002 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.744429 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.744568 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.744626 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.745935 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.746055 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:29:13.746510 10730 leader_election.cc:302] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdd4645b07b84e4c875c9886f306159c, f0a2c92f89b24aea8d135775e09c8c7e; no voters:
I20260810 22:29:13.746613 10505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe773d9cfe424aa80a0a8e4aeac163" candidate_uuid: "cdd4645b07b84e4c875c9886f306159c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e09cde00ff4994a737d17b384928d3" is_pre_election: true
I20260810 22:29:13.746758 10505 raft_consensus.cc:2454] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd4645b07b84e4c875c9886f306159c in term 0.
I20260810 22:29:13.746788 10895 raft_consensus.cc:2781] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:29:13.746840 10895 raft_consensus.cc:482] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:29:13.746876 10895 raft_consensus.cc:3037] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:13.748212 10895 raft_consensus.cc:504] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.748580 10895 leader_election.cc:288] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [CANDIDATE]: Term 1 election: Requested vote from peers d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321), f0a2c92f89b24aea8d135775e09c8c7e (127.1.221.130:36553)
I20260810 22:29:13.748950 10655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe773d9cfe424aa80a0a8e4aeac163" candidate_uuid: "cdd4645b07b84e4c875c9886f306159c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a2c92f89b24aea8d135775e09c8c7e"
I20260810 22:29:13.748979 10505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fe773d9cfe424aa80a0a8e4aeac163" candidate_uuid: "cdd4645b07b84e4c875c9886f306159c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e09cde00ff4994a737d17b384928d3"
I20260810 22:29:13.749096 10505 raft_consensus.cc:3037] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:13.749101 10655 raft_consensus.cc:3037] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:13.750299 10655 raft_consensus.cc:2454] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd4645b07b84e4c875c9886f306159c in term 1.
I20260810 22:29:13.750300 10505 raft_consensus.cc:2454] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd4645b07b84e4c875c9886f306159c in term 1.
I20260810 22:29:13.750609 10731 leader_election.cc:302] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdd4645b07b84e4c875c9886f306159c, d3e09cde00ff4994a737d17b384928d3; no voters:
I20260810 22:29:13.750828 10895 raft_consensus.cc:2781] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:13.751040 10895 raft_consensus.cc:686] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [term 1 LEADER]: Becoming Leader. State: Replica: cdd4645b07b84e4c875c9886f306159c, State: Running, Role: LEADER
I20260810 22:29:13.751323 10895 consensus_queue.cc:237] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.753643 10336 catalog_manager.cc:5495] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c reported cstate change: term changed from 0 to 1, leader changed from <none> to cdd4645b07b84e4c875c9886f306159c (127.1.221.131). New cstate: current_term: 1 leader_uuid: "cdd4645b07b84e4c875c9886f306159c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } health_report { overall_health: HEALTHY } } }
Aug 10 22:29:13 dist-test-slave-njxd krb5kdc[10285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400952, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.190@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260810 22:29:13.789326 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.779521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39405 (local address 127.0.0.1:37494)
0810 22:29:13.779840 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:13.779862 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:13.779968 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:13.780298 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:13.780301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:13.780750 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:13.780859 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.780866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.781985 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.781988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.782544 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:13.782552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.783144 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.783146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:13.783250 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.783868 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:13.783881 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:13.785682 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:13.787501 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:13.787510 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:13.787515 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:13.787742 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:13.788128 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:13.788131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:13.788133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:13.788232 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:13.788700 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:13.788706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:13.788846 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 22:29:13.788972 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 22:29:13.789137 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260810 22:29:13.789675 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:13.779635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.1.221.190:39405)
0810 22:29:13.779790 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:13.779795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:13.779943 (+ 148us) server_negotiation.cc:407] Connection header received
0810 22:29:13.780056 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:13.780059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:13.780124 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:13.780238 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:13.781007 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.781883 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.782662 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:13.783028 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:13.783134 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:13.785823 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:13.785828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:13.785831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:13.785857 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:13.787390 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:13.787850 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:13.787853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:13.787855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:13.787943 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:13.788351 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:13.788355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:13.788357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:13.788576 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:13.788679 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:13.789047 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 22:29:13.789234 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":60}
I20260810 22:29:13.794333 10336 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:37494:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 22:29:13.795204 10336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:29:13.800928 10631 tablet_service.cc:1467] Processing CreateTablet for tablet d3e3ac81d07f44bb86495a37b7bd794d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f79e8a0f7a9f4232b7f0a21d005bc642]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:29:13.800987 10787 tablet_service.cc:1467] Processing CreateTablet for tablet d3e3ac81d07f44bb86495a37b7bd794d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f79e8a0f7a9f4232b7f0a21d005bc642]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:29:13.801110 10484 tablet_service.cc:1467] Processing CreateTablet for tablet d3e3ac81d07f44bb86495a37b7bd794d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f79e8a0f7a9f4232b7f0a21d005bc642]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:29:13.801398 10631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e3ac81d07f44bb86495a37b7bd794d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:13.801395 10787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e3ac81d07f44bb86495a37b7bd794d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:13.801445 10484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e3ac81d07f44bb86495a37b7bd794d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:13.804504 10890 tablet_bootstrap.cc:492] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3: Bootstrap starting.
I20260810 22:29:13.804610 10889 tablet_bootstrap.cc:492] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c: Bootstrap starting.
I20260810 22:29:13.805460 10889 tablet_bootstrap.cc:654] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:13.805460 10890 tablet_bootstrap.cc:654] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:13.805912 10891 tablet_bootstrap.cc:492] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e: Bootstrap starting.
I20260810 22:29:13.806761 10891 tablet_bootstrap.cc:654] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:13.808039 10891 tablet_bootstrap.cc:492] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e: No bootstrap required, opened a new log
I20260810 22:29:13.808130 10891 ts_tablet_manager.cc:1397] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:29:13.808557 10891 raft_consensus.cc:348] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.808673 10891 raft_consensus.cc:374] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:13.808703 10891 raft_consensus.cc:729] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a2c92f89b24aea8d135775e09c8c7e, State: Initialized, Role: FOLLOWER
I20260810 22:29:13.808831 10891 consensus_queue.cc:260] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.809098 10891 ts_tablet_manager.cc:1428] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:29:13.809650 10896 tablet_replica.cc:439] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } }
I20260810 22:29:13.809784 10896 tablet_replica.cc:442] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:29:13.812366 10890 tablet_bootstrap.cc:492] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3: No bootstrap required, opened a new log
I20260810 22:29:13.812471 10890 ts_tablet_manager.cc:1397] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260810 22:29:13.812918 10890 raft_consensus.cc:348] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.813016 10890 raft_consensus.cc:374] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:13.813077 10890 raft_consensus.cc:729] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e09cde00ff4994a737d17b384928d3, State: Initialized, Role: FOLLOWER
I20260810 22:29:13.813222 10890 consensus_queue.cc:260] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.813544 10890 ts_tablet_manager.cc:1428] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:29:13.813476 10897 tablet_replica.cc:439] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } }
I20260810 22:29:13.813659 10897 tablet_replica.cc:442] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:29:13.815243 10889 tablet_bootstrap.cc:492] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c: No bootstrap required, opened a new log
I20260810 22:29:13.815335 10889 ts_tablet_manager.cc:1397] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20260810 22:29:13.815775 10889 raft_consensus.cc:348] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.815878 10889 raft_consensus.cc:374] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:13.815900 10889 raft_consensus.cc:729] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4645b07b84e4c875c9886f306159c, State: Initialized, Role: FOLLOWER
I20260810 22:29:13.816071 10889 consensus_queue.cc:260] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:13.816395 10889 ts_tablet_manager.cc:1428] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:29:13.816334 10895 tablet_replica.cc:439] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } }
I20260810 22:29:13.816473 10895 tablet_replica.cc:442] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:29:14.202432 10895 raft_consensus.cc:482] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:29:14.202611 10895 raft_consensus.cc:504] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.203009 10895 leader_election.cc:288] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321), f0a2c92f89b24aea8d135775e09c8c7e (127.1.221.130:36553)
I20260810 22:29:14.203560 10655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e3ac81d07f44bb86495a37b7bd794d" candidate_uuid: "cdd4645b07b84e4c875c9886f306159c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" is_pre_election: true
I20260810 22:29:14.203563 10505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e3ac81d07f44bb86495a37b7bd794d" candidate_uuid: "cdd4645b07b84e4c875c9886f306159c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e09cde00ff4994a737d17b384928d3" is_pre_election: true
I20260810 22:29:14.203748 10655 raft_consensus.cc:2454] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd4645b07b84e4c875c9886f306159c in term 0.
I20260810 22:29:14.203758 10505 raft_consensus.cc:2454] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd4645b07b84e4c875c9886f306159c in term 0.
I20260810 22:29:14.204103 10730 leader_election.cc:302] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdd4645b07b84e4c875c9886f306159c, f0a2c92f89b24aea8d135775e09c8c7e; no voters:
I20260810 22:29:14.204329 10895 raft_consensus.cc:2781] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:29:14.204408 10895 raft_consensus.cc:482] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:29:14.204479 10895 raft_consensus.cc:3037] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:14.205510 10895 raft_consensus.cc:504] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.205977 10895 leader_election.cc:288] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [CANDIDATE]: Term 1 election: Requested vote from peers d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321), f0a2c92f89b24aea8d135775e09c8c7e (127.1.221.130:36553)
I20260810 22:29:14.206385 10655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e3ac81d07f44bb86495a37b7bd794d" candidate_uuid: "cdd4645b07b84e4c875c9886f306159c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a2c92f89b24aea8d135775e09c8c7e"
I20260810 22:29:14.206406 10505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e3ac81d07f44bb86495a37b7bd794d" candidate_uuid: "cdd4645b07b84e4c875c9886f306159c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e09cde00ff4994a737d17b384928d3"
I20260810 22:29:14.206524 10655 raft_consensus.cc:3037] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:14.206540 10505 raft_consensus.cc:3037] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:14.207448 10505 raft_consensus.cc:2454] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd4645b07b84e4c875c9886f306159c in term 1.
I20260810 22:29:14.207486 10655 raft_consensus.cc:2454] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd4645b07b84e4c875c9886f306159c in term 1.
I20260810 22:29:14.207767 10731 leader_election.cc:302] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdd4645b07b84e4c875c9886f306159c, d3e09cde00ff4994a737d17b384928d3; no voters:
I20260810 22:29:14.207952 10895 raft_consensus.cc:2781] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:14.208046 10895 raft_consensus.cc:686] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [term 1 LEADER]: Becoming Leader. State: Replica: cdd4645b07b84e4c875c9886f306159c, State: Running, Role: LEADER
I20260810 22:29:14.208202 10895 consensus_queue.cc:237] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.208827 10921 tablet_replica.cc:439] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c: TxnStatusTablet state changed. Reason: New leader cdd4645b07b84e4c875c9886f306159c. Latest consensus state: current_term: 1 leader_uuid: "cdd4645b07b84e4c875c9886f306159c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } }
I20260810 22:29:14.208950 10921 tablet_replica.cc:442] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:29:14.209189 10935 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 22:29:14.209390 10935 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 22:29:14.209964 10336 catalog_manager.cc:5495] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c reported cstate change: term changed from 0 to 1, leader changed from <none> to cdd4645b07b84e4c875c9886f306159c (127.1.221.131). New cstate: current_term: 1 leader_uuid: "cdd4645b07b84e4c875c9886f306159c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } health_report { overall_health: HEALTHY } } }
I20260810 22:29:14.221637 10921 consensus_queue.cc:1035] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:14.225679 10921 consensus_queue.cc:1035] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:14.270903 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:14.266194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:32833 (local address 127.0.0.1:33108)
0810 22:29:14.266393 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:14.266413 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:14.266538 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:14.266894 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:14.266896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:14.266902 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:29:14.266996 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:14.267001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.268566 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.268569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:14.269159 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:14.269167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.269776 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.269779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:14.269845 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:14.269890 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:29:14.270657 (+ 767us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:29:14.270668 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 22:29:14.270718 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 22:29:14.270795 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 22:29:14.270948 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:14.266320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.1.221.131:32833)
0810 22:29:14.266603 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:14.266607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:14.266623 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:29:14.266697 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:14.266700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:14.266761 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:14.266877 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:29:14.267104 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.268450 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.269274 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.269657 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.269759 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:14.270030 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:29:14.270559 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:29:14.270672 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:14.270733 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 22:29:14.270788 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":101,"threads_started":1}
I20260810 22:29:14.273821 10505 raft_consensus.cc:1261] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Refusing update from remote peer cdd4645b07b84e4c875c9886f306159c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:14.273854 10655 raft_consensus.cc:1261] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Refusing update from remote peer cdd4645b07b84e4c875c9886f306159c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:14.274323 10895 consensus_queue.cc:1035] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:14.274509 10921 consensus_queue.cc:1035] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:14.276050 10896 tablet_replica.cc:439] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e: TxnStatusTablet state changed. Reason: New leader cdd4645b07b84e4c875c9886f306159c. Latest consensus state: current_term: 1 leader_uuid: "cdd4645b07b84e4c875c9886f306159c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } }
I20260810 22:29:14.276167 10896 tablet_replica.cc:442] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:29:14.276240 10897 tablet_replica.cc:439] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3: TxnStatusTablet state changed. Reason: New leader cdd4645b07b84e4c875c9886f306159c. Latest consensus state: current_term: 1 leader_uuid: "cdd4645b07b84e4c875c9886f306159c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } }
I20260810 22:29:14.276363 10897 tablet_replica.cc:442] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:29:14.278486 10921 tablet_replica.cc:439] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdd4645b07b84e4c875c9886f306159c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } }
I20260810 22:29:14.278609 10921 tablet_replica.cc:442] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:29:14.278694 10921 tablet_replica.cc:439] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdd4645b07b84e4c875c9886f306159c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } }
I20260810 22:29:14.278764 10921 tablet_replica.cc:442] T d3e3ac81d07f44bb86495a37b7bd794d P cdd4645b07b84e4c875c9886f306159c: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:29:14.279242 10897 tablet_replica.cc:439] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdd4645b07b84e4c875c9886f306159c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } }
I20260810 22:29:14.279356 10897 tablet_replica.cc:442] T d3e3ac81d07f44bb86495a37b7bd794d P d3e09cde00ff4994a737d17b384928d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:29:14.280507 10896 tablet_replica.cc:439] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdd4645b07b84e4c875c9886f306159c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } } }
I20260810 22:29:14.280632 10896 tablet_replica.cc:442] T d3e3ac81d07f44bb86495a37b7bd794d P f0a2c92f89b24aea8d135775e09c8c7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:29:14.290184 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:14.286387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33116 (local address 127.1.221.131:32833)
0810 22:29:14.286521 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:14.286525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:14.286700 (+ 175us) server_negotiation.cc:407] Connection header received
0810 22:29:14.286812 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:14.286814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:14.286870 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:14.286931 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:29:14.287222 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.287715 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.288940 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.289145 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.289219 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:14.289509 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:29:14.289631 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:29:14.289717 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:14.290016 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 22:29:14.290070 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:29:14.301388 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:14.297856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:50403 (local address 127.1.221.131:32833)
0810 22:29:14.297965 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:14.297975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:14.298194 (+ 219us) server_negotiation.cc:407] Connection header received
0810 22:29:14.298318 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:14.298323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:14.298371 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:14.298430 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:29:14.298704 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.299279 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.300385 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.300578 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.300648 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:14.300969 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:29:14.301030 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:29:14.301078 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:14.301204 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 22:29:14.301245 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 22:29:14.301446 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:14.297773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:32833 (local address 127.1.221.131:50403)
0810 22:29:14.298111 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:14.298125 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:14.298227 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:14.298468 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:14.298471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:14.298478 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:29:14.298574 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:14.298580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.299365 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.299368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:14.300267 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:14.300278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.300682 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.300686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:14.300771 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:14.300821 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:29:14.301111 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:29:14.301118 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 22:29:14.301187 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 22:29:14.301268 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260810 22:29:14.345940 10337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:14.346149 10337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:14.348672 10337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e: Sending DeleteTablet for 3 replicas of tablet b9fe773d9cfe424aa80a0a8e4aeac163
I20260810 22:29:14.349359 10484 tablet_service.cc:1514] Processing DeleteTablet for tablet b9fe773d9cfe424aa80a0a8e4aeac163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:14 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:46602
I20260810 22:29:14.349468 10631 tablet_service.cc:1514] Processing DeleteTablet for tablet b9fe773d9cfe424aa80a0a8e4aeac163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:14 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:54012
I20260810 22:29:14.349784 10787 tablet_service.cc:1514] Processing DeleteTablet for tablet b9fe773d9cfe424aa80a0a8e4aeac163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:14 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:33102
I20260810 22:29:14.350153 10958 tablet_replica.cc:330] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3: stopping tablet replica
I20260810 22:29:14.350308 10961 tablet_replica.cc:330] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c: stopping tablet replica
I20260810 22:29:14.350346 10958 raft_consensus.cc:2229] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:14.350440 10961 raft_consensus.cc:2229] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:29:14.350641 10958 raft_consensus.cc:2258] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:14.350780 10961 raft_consensus.cc:2258] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:14.351514 10961 ts_tablet_manager.cc:1905] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:14.351516 10958 ts_tablet_manager.cc:1905] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:14.353760 10960 tablet_replica.cc:330] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e: stopping tablet replica
I20260810 22:29:14.354643 10958 ts_tablet_manager.cc:1918] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:29:14.354736 10958 log.cc:1196] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/wal/wals/b9fe773d9cfe424aa80a0a8e4aeac163
I20260810 22:29:14.355028 10958 ts_tablet_manager.cc:1939] T b9fe773d9cfe424aa80a0a8e4aeac163 P d3e09cde00ff4994a737d17b384928d3: Deleting consensus metadata
I20260810 22:29:14.355026 10961 ts_tablet_manager.cc:1918] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:29:14.355099 10961 log.cc:1196] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/b9fe773d9cfe424aa80a0a8e4aeac163
I20260810 22:29:14.355373 10961 ts_tablet_manager.cc:1939] T b9fe773d9cfe424aa80a0a8e4aeac163 P cdd4645b07b84e4c875c9886f306159c: Deleting consensus metadata
I20260810 22:29:14.355902 10321 catalog_manager.cc:4841] TS d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321): tablet b9fe773d9cfe424aa80a0a8e4aeac163 (table test-table [id=737623b13fe9489ea4039d0c2cf3c4f0]) successfully deleted
I20260810 22:29:14.356135 10960 raft_consensus.cc:2229] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:14.356200 10322 catalog_manager.cc:4841] TS cdd4645b07b84e4c875c9886f306159c (127.1.221.131:32833): tablet b9fe773d9cfe424aa80a0a8e4aeac163 (table test-table [id=737623b13fe9489ea4039d0c2cf3c4f0]) successfully deleted
I20260810 22:29:14.356318 10960 raft_consensus.cc:2258] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:14.357108 10960 ts_tablet_manager.cc:1905] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:14.360157 10960 ts_tablet_manager.cc:1918] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:29:14.360258 10960 log.cc:1196] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/b9fe773d9cfe424aa80a0a8e4aeac163
I20260810 22:29:14.360575 10960 ts_tablet_manager.cc:1939] T b9fe773d9cfe424aa80a0a8e4aeac163 P f0a2c92f89b24aea8d135775e09c8c7e: Deleting consensus metadata
I20260810 22:29:14.361264 10320 catalog_manager.cc:4841] TS f0a2c92f89b24aea8d135775e09c8c7e (127.1.221.130:36553): tablet b9fe773d9cfe424aa80a0a8e4aeac163 (table test-table [id=737623b13fe9489ea4039d0c2cf3c4f0]) successfully deleted
Aug 10 22:29:14 dist-test-slave-njxd krb5kdc[10285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:29:14 dist-test-slave-njxd krb5kdc[10285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:14 dist-test-slave-njxd krb5kdc[10285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 22:29:14.382414 10971 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:14.375824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37506 (local address 127.1.221.190:39405)
0810 22:29:14.376068 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:14.376073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:14.376219 (+ 146us) server_negotiation.cc:407] Connection header received
0810 22:29:14.376416 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:14.376420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:14.376468 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:14.376574 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:14.377282 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.377925 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.378808 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.379048 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.379163 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:14.382267 (+ 3104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37506: BlockingRecv error: recv got EOF from 127.0.0.1:37506 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
Aug 10 22:29:14 dist-test-slave-njxd krb5kdc[10285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM
I20260810 22:29:14.393239 10971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:14.383978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.1.221.190:39405)
0810 22:29:14.384117 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:14.384122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:14.384323 (+ 201us) server_negotiation.cc:407] Connection header received
0810 22:29:14.384401 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:14.384404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:14.384452 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:14.384518 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:14.385249 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.385879 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.386960 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.387170 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.387244 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:14.389836 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:14.389840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:14.389843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:14.389871 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:14.391551 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:14.391965 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:14.391968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:14.391970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:14.392070 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:14.392347 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:14.392350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:14.392352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:14.392579 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:14.392719 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:14.392945 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 22:29:14.393070 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:29:14.395977 10337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:29:14.396452 10337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:29:14.402101 10787 tablet_service.cc:1467] Processing CreateTablet for tablet 1291e81ad1214af494b49f4d32475e4e (DEFAULT_TABLE table=test-table [id=03db5455cc43426faea7276d6dbc2d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:14.402220 10484 tablet_service.cc:1467] Processing CreateTablet for tablet 1291e81ad1214af494b49f4d32475e4e (DEFAULT_TABLE table=test-table [id=03db5455cc43426faea7276d6dbc2d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:14.402256 10631 tablet_service.cc:1467] Processing CreateTablet for tablet 1291e81ad1214af494b49f4d32475e4e (DEFAULT_TABLE table=test-table [id=03db5455cc43426faea7276d6dbc2d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:14.402441 10787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1291e81ad1214af494b49f4d32475e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:14.402482 10484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1291e81ad1214af494b49f4d32475e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:14.402567 10631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1291e81ad1214af494b49f4d32475e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:14.405465 10979 tablet_bootstrap.cc:492] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c: Bootstrap starting.
I20260810 22:29:14.406456 10979 tablet_bootstrap.cc:654] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:14.407117 10980 tablet_bootstrap.cc:492] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3: Bootstrap starting.
I20260810 22:29:14.407689 10979 tablet_bootstrap.cc:492] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c: No bootstrap required, opened a new log
I20260810 22:29:14.407778 10979 ts_tablet_manager.cc:1397] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:29:14.408097 10980 tablet_bootstrap.cc:654] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:14.408175 10979 raft_consensus.cc:348] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.408282 10979 raft_consensus.cc:374] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:14.408334 10979 raft_consensus.cc:729] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4645b07b84e4c875c9886f306159c, State: Initialized, Role: FOLLOWER
I20260810 22:29:14.408481 10979 consensus_queue.cc:260] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.408818 10981 tablet_bootstrap.cc:492] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e: Bootstrap starting.
I20260810 22:29:14.409276 10979 ts_tablet_manager.cc:1428] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:29:14.410265 10981 tablet_bootstrap.cc:654] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:14.410449 10980 tablet_bootstrap.cc:492] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3: No bootstrap required, opened a new log
I20260810 22:29:14.410528 10980 ts_tablet_manager.cc:1397] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:29:14.410989 10980 raft_consensus.cc:348] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.411114 10980 raft_consensus.cc:374] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:14.411156 10980 raft_consensus.cc:729] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e09cde00ff4994a737d17b384928d3, State: Initialized, Role: FOLLOWER
I20260810 22:29:14.411345 10980 consensus_queue.cc:260] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.412055 10980 ts_tablet_manager.cc:1428] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:29:14.414390 10981 tablet_bootstrap.cc:492] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e: No bootstrap required, opened a new log
I20260810 22:29:14.414505 10981 ts_tablet_manager.cc:1397] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260810 22:29:14.415035 10981 raft_consensus.cc:348] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.415150 10981 raft_consensus.cc:374] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:14.415196 10981 raft_consensus.cc:729] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a2c92f89b24aea8d135775e09c8c7e, State: Initialized, Role: FOLLOWER
I20260810 22:29:14.415359 10981 consensus_queue.cc:260] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.415652 10981 ts_tablet_manager.cc:1428] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:29:14.422482 10896 raft_consensus.cc:482] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:29:14.422600 10896 raft_consensus.cc:504] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.422989 10896 leader_election.cc:288] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321), cdd4645b07b84e4c875c9886f306159c (127.1.221.131:32833)
I20260810 22:29:14.423413 10811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1291e81ad1214af494b49f4d32475e4e" candidate_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4645b07b84e4c875c9886f306159c" is_pre_election: true
I20260810 22:29:14.423434 10505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1291e81ad1214af494b49f4d32475e4e" candidate_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e09cde00ff4994a737d17b384928d3" is_pre_election: true
I20260810 22:29:14.423571 10811 raft_consensus.cc:2454] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a2c92f89b24aea8d135775e09c8c7e in term 0.
I20260810 22:29:14.423588 10505 raft_consensus.cc:2454] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a2c92f89b24aea8d135775e09c8c7e in term 0.
I20260810 22:29:14.423910 10580 leader_election.cc:302] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3e09cde00ff4994a737d17b384928d3, f0a2c92f89b24aea8d135775e09c8c7e; no voters:
I20260810 22:29:14.424126 10896 raft_consensus.cc:2781] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:29:14.424177 10896 raft_consensus.cc:482] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:29:14.424220 10896 raft_consensus.cc:3037] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:14.425410 10896 raft_consensus.cc:504] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.425772 10896 leader_election.cc:288] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [CANDIDATE]: Term 1 election: Requested vote from peers d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321), cdd4645b07b84e4c875c9886f306159c (127.1.221.131:32833)
I20260810 22:29:14.426106 10505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1291e81ad1214af494b49f4d32475e4e" candidate_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e09cde00ff4994a737d17b384928d3"
I20260810 22:29:14.426254 10505 raft_consensus.cc:3037] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:14.426283 10811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1291e81ad1214af494b49f4d32475e4e" candidate_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4645b07b84e4c875c9886f306159c"
I20260810 22:29:14.426407 10811 raft_consensus.cc:3037] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:14.427170 10505 raft_consensus.cc:2454] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a2c92f89b24aea8d135775e09c8c7e in term 1.
I20260810 22:29:14.427456 10580 leader_election.cc:302] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3e09cde00ff4994a737d17b384928d3, f0a2c92f89b24aea8d135775e09c8c7e; no voters:
I20260810 22:29:14.427502 10811 raft_consensus.cc:2454] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a2c92f89b24aea8d135775e09c8c7e in term 1.
I20260810 22:29:14.427678 10896 raft_consensus.cc:2781] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:14.427858 10896 raft_consensus.cc:686] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [term 1 LEADER]: Becoming Leader. State: Replica: f0a2c92f89b24aea8d135775e09c8c7e, State: Running, Role: LEADER
I20260810 22:29:14.428020 10896 consensus_queue.cc:237] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.429679 10336 catalog_manager.cc:5495] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e reported cstate change: term changed from 0 to 1, leader changed from <none> to f0a2c92f89b24aea8d135775e09c8c7e (127.1.221.130). New cstate: current_term: 1 leader_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } health_report { overall_health: UNKNOWN } } }
I20260810 22:29:14.443795 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:14.440172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.1.221.130:36553)
0810 22:29:14.440429 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:14.440436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:14.440448 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:29:14.440486 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:14.440488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:14.440527 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:14.440628 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:29:14.440906 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.441766 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.442611 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.442985 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.443095 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:14.443310 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:29:14.443399 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:29:14.443473 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:14.443580 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 22:29:14.443639 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":119,"threads_started":1}
I20260810 22:29:14.445820 10505 raft_consensus.cc:1261] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Refusing update from remote peer f0a2c92f89b24aea8d135775e09c8c7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:14.445832 10811 raft_consensus.cc:1261] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Refusing update from remote peer f0a2c92f89b24aea8d135775e09c8c7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:14.446265 10986 consensus_queue.cc:1035] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:14.446465 10896 consensus_queue.cc:1035] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:14.450402 10941 mvcc.cc:205] Tried to move back new op lower bound from 7317098309405560832 to 7317098309338574848. Current Snapshot: MvccSnapshot[applied={T|T < 7317098309405560832}]
I20260810 22:29:14.454618 10336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:14.454789 10336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:14.456889 10336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e: Sending DeleteTablet for 3 replicas of tablet 1291e81ad1214af494b49f4d32475e4e
I20260810 22:29:14.457492 10484 tablet_service.cc:1514] Processing DeleteTablet for tablet 1291e81ad1214af494b49f4d32475e4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:14 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:46602
I20260810 22:29:14.457691 10958 tablet_replica.cc:330] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3: stopping tablet replica
I20260810 22:29:14.457830 10958 raft_consensus.cc:2229] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:14.457995 10958 raft_consensus.cc:2258] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:14.458248 10336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:29:14.458607 10631 tablet_service.cc:1514] Processing DeleteTablet for tablet 1291e81ad1214af494b49f4d32475e4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:14 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:54012
I20260810 22:29:14.458600 10787 tablet_service.cc:1514] Processing DeleteTablet for tablet 1291e81ad1214af494b49f4d32475e4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:14 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:33102
W20260810 22:29:14.458647 10336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:29:14.458784 10961 tablet_replica.cc:330] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c: stopping tablet replica
I20260810 22:29:14.458777 10960 tablet_replica.cc:330] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e: stopping tablet replica
I20260810 22:29:14.458884 10960 raft_consensus.cc:2229] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:29:14.458900 10961 raft_consensus.cc:2229] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:14.459009 10961 raft_consensus.cc:2258] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:14.459056 10960 raft_consensus.cc:2258] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:14.459847 10961 ts_tablet_manager.cc:1905] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:14.462535 10961 ts_tablet_manager.cc:1918] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:14.462644 10961 log.cc:1196] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/1291e81ad1214af494b49f4d32475e4e
I20260810 22:29:14.462947 10961 ts_tablet_manager.cc:1939] T 1291e81ad1214af494b49f4d32475e4e P cdd4645b07b84e4c875c9886f306159c: Deleting consensus metadata
I20260810 22:29:14.463570 10631 tablet_service.cc:1467] Processing CreateTablet for tablet 488daa1b6e78408990600679f984fbd9 (DEFAULT_TABLE table=test-table [id=7c86d4a5cc9f4fe6b84bbdc706a1e7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:14.463721 10787 tablet_service.cc:1467] Processing CreateTablet for tablet 488daa1b6e78408990600679f984fbd9 (DEFAULT_TABLE table=test-table [id=7c86d4a5cc9f4fe6b84bbdc706a1e7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:14.463833 10631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488daa1b6e78408990600679f984fbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:14.463930 10787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488daa1b6e78408990600679f984fbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:14.466001 10960 ts_tablet_manager.cc:1905] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:14.466502 10484 tablet_service.cc:1467] Processing CreateTablet for tablet 488daa1b6e78408990600679f984fbd9 (DEFAULT_TABLE table=test-table [id=7c86d4a5cc9f4fe6b84bbdc706a1e7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:14.466737 10484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 488daa1b6e78408990600679f984fbd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:14.468176 10960 ts_tablet_manager.cc:1918] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:14.468250 10960 log.cc:1196] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/1291e81ad1214af494b49f4d32475e4e
I20260810 22:29:14.468510 10960 ts_tablet_manager.cc:1939] T 1291e81ad1214af494b49f4d32475e4e P f0a2c92f89b24aea8d135775e09c8c7e: Deleting consensus metadata
I20260810 22:29:14.469110 10981 tablet_bootstrap.cc:492] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e: Bootstrap starting.
I20260810 22:29:14.469276 10320 catalog_manager.cc:4841] TS f0a2c92f89b24aea8d135775e09c8c7e (127.1.221.130:36553): tablet 1291e81ad1214af494b49f4d32475e4e (table test-table [id=03db5455cc43426faea7276d6dbc2d6c]) successfully deleted
I20260810 22:29:14.469702 10980 tablet_bootstrap.cc:492] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3: Bootstrap starting.
I20260810 22:29:14.470003 10981 tablet_bootstrap.cc:654] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:14.470158 10979 tablet_bootstrap.cc:492] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c: Bootstrap starting.
I20260810 22:29:14.470913 10980 tablet_bootstrap.cc:654] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:14.470976 10979 tablet_bootstrap.cc:654] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:14.471320 10981 tablet_bootstrap.cc:492] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e: No bootstrap required, opened a new log
I20260810 22:29:14.471402 10981 ts_tablet_manager.cc:1397] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:29:14.471602 10322 catalog_manager.cc:4841] TS cdd4645b07b84e4c875c9886f306159c (127.1.221.131:32833): tablet 1291e81ad1214af494b49f4d32475e4e (table test-table [id=03db5455cc43426faea7276d6dbc2d6c]) successfully deleted
I20260810 22:29:14.471828 10981 raft_consensus.cc:348] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.471952 10981 raft_consensus.cc:374] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:14.471990 10981 raft_consensus.cc:729] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a2c92f89b24aea8d135775e09c8c7e, State: Initialized, Role: FOLLOWER
I20260810 22:29:14.472132 10981 consensus_queue.cc:260] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.472285 10958 ts_tablet_manager.cc:1905] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:14.472411 10981 ts_tablet_manager.cc:1428] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:29:14.473410 10980 tablet_bootstrap.cc:492] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3: No bootstrap required, opened a new log
I20260810 22:29:14.473495 10980 ts_tablet_manager.cc:1397] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 22:29:14.473927 10980 raft_consensus.cc:348] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.474026 10980 raft_consensus.cc:374] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:14.474053 10980 raft_consensus.cc:729] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e09cde00ff4994a737d17b384928d3, State: Initialized, Role: FOLLOWER
I20260810 22:29:14.474154 10980 consensus_queue.cc:260] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.475003 10979 tablet_bootstrap.cc:492] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c: No bootstrap required, opened a new log
I20260810 22:29:14.475085 10979 ts_tablet_manager.cc:1397] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260810 22:29:14.475199 10980 ts_tablet_manager.cc:1428] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 22:29:14.475503 10979 raft_consensus.cc:348] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.475625 10979 raft_consensus.cc:374] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:14.475659 10979 raft_consensus.cc:729] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4645b07b84e4c875c9886f306159c, State: Initialized, Role: FOLLOWER
I20260810 22:29:14.475785 10979 consensus_queue.cc:260] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.476071 10979 ts_tablet_manager.cc:1428] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:29:14.477495 10958 ts_tablet_manager.cc:1918] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:14.477605 10958 log.cc:1196] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-0/wal/wals/1291e81ad1214af494b49f4d32475e4e
I20260810 22:29:14.477895 10958 ts_tablet_manager.cc:1939] T 1291e81ad1214af494b49f4d32475e4e P d3e09cde00ff4994a737d17b384928d3: Deleting consensus metadata
I20260810 22:29:14.478484 10321 catalog_manager.cc:4841] TS d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321): tablet 1291e81ad1214af494b49f4d32475e4e (table test-table [id=03db5455cc43426faea7276d6dbc2d6c]) successfully deleted
I20260810 22:29:14.687139 10986 raft_consensus.cc:482] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:29:14.687341 10986 raft_consensus.cc:504] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.687839 10986 leader_election.cc:288] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321), cdd4645b07b84e4c875c9886f306159c (127.1.221.131:32833)
I20260810 22:29:14.688337 10505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488daa1b6e78408990600679f984fbd9" candidate_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e09cde00ff4994a737d17b384928d3" is_pre_election: true
I20260810 22:29:14.688342 10811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488daa1b6e78408990600679f984fbd9" candidate_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4645b07b84e4c875c9886f306159c" is_pre_election: true
I20260810 22:29:14.688525 10505 raft_consensus.cc:2454] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a2c92f89b24aea8d135775e09c8c7e in term 0.
I20260810 22:29:14.688534 10811 raft_consensus.cc:2454] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0a2c92f89b24aea8d135775e09c8c7e in term 0.
I20260810 22:29:14.688931 10580 leader_election.cc:302] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3e09cde00ff4994a737d17b384928d3, f0a2c92f89b24aea8d135775e09c8c7e; no voters:
I20260810 22:29:14.689126 10986 raft_consensus.cc:2781] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:29:14.689188 10986 raft_consensus.cc:482] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:29:14.689215 10986 raft_consensus.cc:3037] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:14.690299 10986 raft_consensus.cc:504] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.690631 10986 leader_election.cc:288] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [CANDIDATE]: Term 1 election: Requested vote from peers d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321), cdd4645b07b84e4c875c9886f306159c (127.1.221.131:32833)
I20260810 22:29:14.690990 10505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488daa1b6e78408990600679f984fbd9" candidate_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3e09cde00ff4994a737d17b384928d3"
I20260810 22:29:14.691025 10811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "488daa1b6e78408990600679f984fbd9" candidate_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4645b07b84e4c875c9886f306159c"
I20260810 22:29:14.691105 10505 raft_consensus.cc:3037] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:14.691165 10811 raft_consensus.cc:3037] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:14.692006 10505 raft_consensus.cc:2454] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a2c92f89b24aea8d135775e09c8c7e in term 1.
I20260810 22:29:14.692209 10811 raft_consensus.cc:2454] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0a2c92f89b24aea8d135775e09c8c7e in term 1.
I20260810 22:29:14.692366 10580 leader_election.cc:302] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3e09cde00ff4994a737d17b384928d3, f0a2c92f89b24aea8d135775e09c8c7e; no voters:
I20260810 22:29:14.692619 10986 raft_consensus.cc:2781] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:14.692731 10986 raft_consensus.cc:686] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [term 1 LEADER]: Becoming Leader. State: Replica: f0a2c92f89b24aea8d135775e09c8c7e, State: Running, Role: LEADER
I20260810 22:29:14.692880 10986 consensus_queue.cc:237] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } }
I20260810 22:29:14.694686 10336 catalog_manager.cc:5495] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e reported cstate change: term changed from 0 to 1, leader changed from <none> to f0a2c92f89b24aea8d135775e09c8c7e (127.1.221.130). New cstate: current_term: 1 leader_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a2c92f89b24aea8d135775e09c8c7e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 } health_report { overall_health: UNKNOWN } } }
I20260810 22:29:14.714365 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:14.711392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:45431 (local address 127.1.221.131:32833)
0810 22:29:14.711513 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:14.711517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:14.711677 (+ 160us) server_negotiation.cc:407] Connection header received
0810 22:29:14.711770 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:14.711773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:14.711815 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:14.711885 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:29:14.712106 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.712601 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.713410 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.713685 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.713762 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:14.714016 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:29:14.714079 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:29:14.714130 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:14.714218 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 22:29:14.714259 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:29:14.714433 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:14.711284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:32833 (local address 127.1.221.130:45431)
0810 22:29:14.711597 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:14.711611 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:14.711702 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:14.711913 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:14.711915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:14.711921 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:29:14.712012 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:14.712016 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.712691 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.712693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:14.713308 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:14.713315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.713785 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.713793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:14.713859 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:14.713902 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:29:14.714159 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:29:14.714164 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 22:29:14.714211 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 22:29:14.714279 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":123,"threads_started":1}
I20260810 22:29:14.719988 10811 raft_consensus.cc:1261] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Refusing update from remote peer f0a2c92f89b24aea8d135775e09c8c7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:14.719988 10505 raft_consensus.cc:1261] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Refusing update from remote peer f0a2c92f89b24aea8d135775e09c8c7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:14.720414 10896 consensus_queue.cc:1035] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4645b07b84e4c875c9886f306159c" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:14.720573 10986 consensus_queue.cc:1035] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3e09cde00ff4994a737d17b384928d3" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:14.738168 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:14.734506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36553 (local address 127.1.221.131:55797)
0810 22:29:14.734706 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:14.734722 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:14.734809 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:14.735094 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:14.735097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:14.735105 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:29:14.735207 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:14.735213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.736252 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.736254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:14.736839 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:14.736847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.737291 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.737297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:14.737386 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:14.737413 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:29:14.737816 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:29:14.737824 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:29:14.737892 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 22:29:14.738001 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260810 22:29:14.738226 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:14.734626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:55797 (local address 127.1.221.130:36553)
0810 22:29:14.734736 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:14.734741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:14.734759 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:29:14.734921 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:14.734924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:14.734979 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:14.735063 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:29:14.735318 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.736133 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.736943 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:14.737173 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:14.737261 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:14.737618 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:29:14.737720 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:29:14.737794 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:14.738039 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 22:29:14.738101 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 22:29:14.759438 10336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:14.759588 10336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:14.761718 10336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3736b230c488481b8dfeb8f9e17acd6e: Sending DeleteTablet for 3 replicas of tablet 488daa1b6e78408990600679f984fbd9
I20260810 22:29:14.762351 10631 tablet_service.cc:1514] Processing DeleteTablet for tablet 488daa1b6e78408990600679f984fbd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:14 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:54012
I20260810 22:29:14.762411 10484 tablet_service.cc:1514] Processing DeleteTablet for tablet 488daa1b6e78408990600679f984fbd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:14 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:46602
I20260810 22:29:14.762562 10960 tablet_replica.cc:330] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e: stopping tablet replica
I20260810 22:29:14.762603 10958 tablet_replica.cc:330] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3: stopping tablet replica
I20260810 22:29:14.762588 10787 tablet_service.cc:1514] Processing DeleteTablet for tablet 488daa1b6e78408990600679f984fbd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:14 UTC) from {username='oryx', principal='oryx/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:33102
I20260810 22:29:14.762687 10960 raft_consensus.cc:2229] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:29:14.762722 10958 raft_consensus.cc:2229] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:14.762778 10961 tablet_replica.cc:330] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c: stopping tablet replica
I20260810 22:29:14.762828 10958 raft_consensus.cc:2258] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:14.762895 10961 raft_consensus.cc:2229] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:14.762908 10960 raft_consensus.cc:2258] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:14.763024 10961 raft_consensus.cc:2258] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:14.763598 10958 ts_tablet_manager.cc:1905] T 488daa1b6e78408990600679f984fbd9 P d3e09cde00ff4994a737d17b384928d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:14.763768 10960 ts_tablet_manager.cc:1905] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:14.763833 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 10413
I20260810 22:29:14.763880 10961 ts_tablet_manager.cc:1905] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:14.766574 10961 ts_tablet_manager.cc:1918] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:29:14.766659 10961 log.cc:1196] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/488daa1b6e78408990600679f984fbd9
I20260810 22:29:14.766916 10961 ts_tablet_manager.cc:1939] T 488daa1b6e78408990600679f984fbd9 P cdd4645b07b84e4c875c9886f306159c: Deleting consensus metadata
I20260810 22:29:14.767627 10322 catalog_manager.cc:4841] TS cdd4645b07b84e4c875c9886f306159c (127.1.221.131:32833): tablet 488daa1b6e78408990600679f984fbd9 (table test-table [id=7c86d4a5cc9f4fe6b84bbdc706a1e7b6]) successfully deleted
I20260810 22:29:14.767901 10960 ts_tablet_manager.cc:1918] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:29:14.768002 10960 log.cc:1196] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/488daa1b6e78408990600679f984fbd9
I20260810 22:29:14.768317 10960 ts_tablet_manager.cc:1939] T 488daa1b6e78408990600679f984fbd9 P f0a2c92f89b24aea8d135775e09c8c7e: Deleting consensus metadata
I20260810 22:29:14.768985 10320 catalog_manager.cc:4841] TS f0a2c92f89b24aea8d135775e09c8c7e (127.1.221.130:36553): tablet 488daa1b6e78408990600679f984fbd9 (table test-table [id=7c86d4a5cc9f4fe6b84bbdc706a1e7b6]) successfully deleted
W20260810 22:29:14.773283 10321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:36321 (error 108)
I20260810 22:29:14.774103 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 10563
I20260810 22:29:14.775755 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:14.774060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:36321 (local address 127.0.0.1:46604)
0810 22:29:14.775596 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:14.775668 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1434,"thread_start_us":77,"threads_started":1}
W20260810 22:29:14.775957 10321 catalog_manager.cc:4584] TS d3e09cde00ff4994a737d17b384928d3 (127.1.221.129:36321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 488daa1b6e78408990600679f984fbd9: Network error: Client connection negotiation failed: client connection to 127.1.221.129:36321: connect: Connection refused (error 111)
I20260810 22:29:14.783684 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 10714
I20260810 22:29:14.793818 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 10302
2026-08-10T22:29:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3738 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:29:14 dist-test-slave-njxd krb5kdc[11008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:29:14 dist-test-slave-njxd krb5kdc[11008](info): set up 2 sockets
Aug 10 22:29:14 dist-test-slave-njxd krb5kdc[11008](info): commencing operation
krb5kdc: starting...
W20260810 22:29:15.895710 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:29:15 dist-test-slave-njxd krb5kdc[11008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:29:16.024967 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42503
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:42503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:16.136830 11022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:16.137068 11022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:16.137106 11022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:16.140878 11022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:16.140931 11022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:16.140949 11022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:16.140965 11022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:16.140980 11022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:29:16.145973 11022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:42503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:16.146896 11022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:16.147835 11022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:16.156113 11028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:16.156155 11027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:16.156137 11030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:16.156314 11022 server_base.cc:1034] running on GCE node
I20260810 22:29:16.157095 11022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:16.158270 11022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:16.159437 11022 hybrid_clock.cc:648] HybridClock initialized: now 1786400956159409 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T22:29:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1372 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:29:16 dist-test-slave-njxd krb5kdc[11036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:29:16 dist-test-slave-njxd krb5kdc[11036](info): set up 2 sockets
Aug 10 22:29:16 dist-test-slave-njxd krb5kdc[11036](info): commencing operation
krb5kdc: starting...
W20260810 22:29:17.390900 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:29:17 dist-test-slave-njxd krb5kdc[11036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:29:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:29:17.587931 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:35453
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:35453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:17.775529 11049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:17.775969 11049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:17.776083 11049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:17.781108 11049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:17.781226 11049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:17.781311 11049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:17.781391 11049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:17.781455 11049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:29:17.788295 11049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:35453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:35453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:17.789839 11049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:17.791414 11049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:17.803802 11054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:17.803967 11055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:17.803807 11058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:17.804446 11049 server_base.cc:1034] running on GCE node
I20260810 22:29:17.805271 11049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:17.806951 11049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:17.808270 11049 hybrid_clock.cc:648] HybridClock initialized: now 1786400957808253 us; error 45 us; skew 500 ppm
Aug 10 22:29:17 dist-test-slave-njxd krb5kdc[11036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400957, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:17.815640 11049 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:29:17.817080 11049 webserver.cc:460] Webserver started at http://127.1.221.190:42695/ using document root <none> and password file <none>
I20260810 22:29:17.817935 11049 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:17.818035 11049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:17.818301 11049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:17.821224 11049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "2b4a79b4181d4cf2b5afbd7291673b65"
format_stamp: "Formatted at 2026-08-10 22:29:17 on dist-test-slave-njxd"
server_key: "308548df24b4c6c3da33f50f730a41af"
server_key_iv: "c629400e8471ee9e9b4f41b7dbcfd93d"
server_key_version: "encryptionkey@0"
I20260810 22:29:17.822067 11049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "2b4a79b4181d4cf2b5afbd7291673b65"
format_stamp: "Formatted at 2026-08-10 22:29:17 on dist-test-slave-njxd"
server_key: "308548df24b4c6c3da33f50f730a41af"
server_key_iv: "c629400e8471ee9e9b4f41b7dbcfd93d"
server_key_version: "encryptionkey@0"
I20260810 22:29:17.827565 11049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 22:29:17.831137 11067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:17.832857 11049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:29:17.833037 11049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "2b4a79b4181d4cf2b5afbd7291673b65"
format_stamp: "Formatted at 2026-08-10 22:29:17 on dist-test-slave-njxd"
server_key: "308548df24b4c6c3da33f50f730a41af"
server_key_iv: "c629400e8471ee9e9b4f41b7dbcfd93d"
server_key_version: "encryptionkey@0"
I20260810 22:29:17.833179 11049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:17.855854 11049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:17.860692 11049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:17.861045 11049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:17.872915 11049 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:35453
I20260810 22:29:17.872953 11127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:35453 every 8 connection(s)
I20260810 22:29:17.874500 11049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:29:17.879092 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 11049
I20260810 22:29:17.879168 11128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:17.879328 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:29:17.879940 1910 external_mini_cluster.cc:1428] Setting key 1aaf62f50e9eece9f019df2559206b85
I20260810 22:29:17.890439 11128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65: Bootstrap starting.
Aug 10 22:29:17 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:29:17.895033 11128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:17.896221 11128 log.cc:824] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:17.899191 11128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65: No bootstrap required, opened a new log
I20260810 22:29:17.903795 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:17.882210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56440 (local address 127.1.221.190:35453)
0810 22:29:17.882894 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:17.882923 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:17.882981 (+ 58us) server_negotiation.cc:407] Connection header received
0810 22:29:17.884060 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:17.884088 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:17.884610 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:17.885051 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:17.886653 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:17.888096 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:17.889756 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:17.890484 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:17.890765 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:17.895260 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:17.895272 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:17.895288 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:17.895337 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:17.898099 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:17.898839 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:17.898851 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:17.898858 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:17.899034 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:17.899515 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:17.899525 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:17.899528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:17.900435 (+ 907us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:17.900767 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:17.901234 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 22:29:17.901773 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":227,"threads_started":1}
I20260810 22:29:17.906232 11128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35453 } }
I20260810 22:29:17.906595 11128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:17.906701 11128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4a79b4181d4cf2b5afbd7291673b65, State: Initialized, Role: FOLLOWER
I20260810 22:29:17.907433 11128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35453 } }
I20260810 22:29:17.907647 11128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:29:17.907747 11128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:29:17.907922 11128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:17.909314 11128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35453 } }
I20260810 22:29:17.909848 11128 leader_election.cc:302] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b4a79b4181d4cf2b5afbd7291673b65; no voters:
I20260810 22:29:17.910323 11128 leader_election.cc:288] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:29:17.910506 11133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:17.910863 11133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 1 LEADER]: Becoming Leader. State: Replica: 2b4a79b4181d4cf2b5afbd7291673b65, State: Running, Role: LEADER
I20260810 22:29:17.911375 11133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35453 } }
I20260810 22:29:17.912206 11128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:17.914141 11135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b4a79b4181d4cf2b5afbd7291673b65. Latest consensus state: current_term: 1 leader_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35453 } } }
I20260810 22:29:17.914100 11134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35453 } } }
I20260810 22:29:17.914333 11134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:17.914333 11135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:17.914983 11142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:17.920423 11142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:17.932236 11142 catalog_manager.cc:1347] Generated new cluster ID: 839c663a939645349ec7889ca6a5cc72
I20260810 22:29:17.932363 11142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:17.940018 11142 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:29:17.941351 11142 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:17.952713 11142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65: Generated new TSK 0
I20260810 22:29:17.954233 11142 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:29:17.993999 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:35453
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:18.186252 11157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:18.186791 11157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:18.187005 11157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:18.193252 11157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:18.193410 11157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:18.193634 11157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:29:18.202963 11157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:35453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:18.204854 11157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:18.206681 11157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:18.223570 11165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:18.223548 11162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:18.223573 11163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:18.223937 11157 server_base.cc:1034] running on GCE node
I20260810 22:29:18.224866 11157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:18.225893 11157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:18.227119 11157 hybrid_clock.cc:648] HybridClock initialized: now 1786400958227083 us; error 48 us; skew 500 ppm
Aug 10 22:29:18 dist-test-slave-njxd krb5kdc[11036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400958, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:18.231917 11157 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:29:18.233354 11157 webserver.cc:460] Webserver started at http://127.1.221.129:36131/ using document root <none> and password file <none>
I20260810 22:29:18.234331 11157 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:18.234472 11157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:18.234890 11157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:18.238541 11157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "1b98ed55c17b428bbe34984570f07372"
format_stamp: "Formatted at 2026-08-10 22:29:18 on dist-test-slave-njxd"
server_key: "f3a8ff794480df7f1136e5dc07e9b13e"
server_key_iv: "f6baa9bc045ae7cfd9f2655915520e8e"
server_key_version: "encryptionkey@0"
I20260810 22:29:18.239612 11157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "1b98ed55c17b428bbe34984570f07372"
format_stamp: "Formatted at 2026-08-10 22:29:18 on dist-test-slave-njxd"
server_key: "f3a8ff794480df7f1136e5dc07e9b13e"
server_key_iv: "f6baa9bc045ae7cfd9f2655915520e8e"
server_key_version: "encryptionkey@0"
I20260810 22:29:18.246225 11157 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 22:29:18.250449 11173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:18.252593 11157 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:29:18.252806 11157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "1b98ed55c17b428bbe34984570f07372"
format_stamp: "Formatted at 2026-08-10 22:29:18 on dist-test-slave-njxd"
server_key: "f3a8ff794480df7f1136e5dc07e9b13e"
server_key_iv: "f6baa9bc045ae7cfd9f2655915520e8e"
server_key_version: "encryptionkey@0"
I20260810 22:29:18.252971 11157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:18.302964 11157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:18.308076 11157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:18.308471 11157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:18.309695 11157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:18.311070 11157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:18.311139 11157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:18.311277 11157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:18.311345 11157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:18.328101 11157 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:38485
I20260810 22:29:18.328122 11294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:38485 every 8 connection(s)
I20260810 22:29:18.329767 11157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:29:18.337021 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 11157
I20260810 22:29:18.337230 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:29:18.337831 1910 external_mini_cluster.cc:1428] Setting key d982d5536eaaf5553b1ccff62dc39b14
Aug 10 22:29:18 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400958, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260810 22:29:18.353685 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:18.332701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:43195 (local address 127.1.221.190:35453)
0810 22:29:18.332906 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:18.332911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:18.334618 (+ 1707us) server_negotiation.cc:407] Connection header received
0810 22:29:18.336844 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:18.336849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:18.336995 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:18.337198 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:18.339770 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:18.340565 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:18.342602 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:18.342961 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:18.343189 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:18.347552 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:18.347556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:18.347558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:18.347600 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:18.350052 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:18.350974 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:18.350994 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:18.350996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:18.351109 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:18.351731 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:18.351735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:18.351736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:18.352033 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:18.352229 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:18.353142 (+ 913us) server_negotiation.cc:299] Negotiation successful
0810 22:29:18.353378 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 22:29:18.355427 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:18.333491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:43195)
0810 22:29:18.334355 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:18.334446 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:18.336449 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:18.337556 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:18.337619 (+ 63us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:18.338369 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:18.339536 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:18.339562 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:18.340798 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:18.340802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:18.342377 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:18.342388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:18.343269 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:18.343275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:18.343585 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:18.344652 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:18.344682 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:18.347259 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:18.350291 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:18.350310 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:18.350338 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:18.350786 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:18.351302 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:18.351305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:18.351307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:18.351560 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:18.352247 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:18.352259 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:18.352693 (+ 434us) client_negotiation.cc:769] Sending connection context
0810 22:29:18.353048 (+ 355us) client_negotiation.cc:240] Negotiation successful
0810 22:29:18.353709 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":196,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260810 22:29:18.357481 11295 heartbeater.cc:347] Connected to a master server at 127.1.221.190:35453
I20260810 22:29:18.357897 11295 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:18.358649 11295 heartbeater.cc:511] Master 127.1.221.190:35453 requested a full tablet report, sending...
I20260810 22:29:18.361012 11087 ts_manager.cc:194] Registered new tserver with Master: 1b98ed55c17b428bbe34984570f07372 (127.1.221.129:38485)
I20260810 22:29:18.362774 11087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:43195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:29:18.379355 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:35453
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:18.502403 11300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:18.502693 11300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:18.502784 11300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:18.506953 11300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:18.507017 11300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:18.507115 11300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:29:18.512876 11300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:35453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:18.514096 11300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:18.515226 11300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:18.525009 11306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:18.525077 11309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:18.525076 11307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:18.525288 11300 server_base.cc:1034] running on GCE node
I20260810 22:29:18.525954 11300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:18.526646 11300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:18.527843 11300 hybrid_clock.cc:648] HybridClock initialized: now 1786400958527830 us; error 31 us; skew 500 ppm
Aug 10 22:29:18 dist-test-slave-njxd krb5kdc[11036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400958, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:18.530882 11300 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:29:18.531821 11300 webserver.cc:460] Webserver started at http://127.1.221.130:33365/ using document root <none> and password file <none>
I20260810 22:29:18.532428 11300 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:18.532526 11300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:18.532773 11300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:18.535034 11300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "2d367e098c5f47eabc8dc2c9f45f718d"
format_stamp: "Formatted at 2026-08-10 22:29:18 on dist-test-slave-njxd"
server_key: "dcbe35041ee4fc38dcacf3787c3d2888"
server_key_iv: "964dc2d3085187d8ace5f484bd1debe1"
server_key_version: "encryptionkey@0"
I20260810 22:29:18.535653 11300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "2d367e098c5f47eabc8dc2c9f45f718d"
format_stamp: "Formatted at 2026-08-10 22:29:18 on dist-test-slave-njxd"
server_key: "dcbe35041ee4fc38dcacf3787c3d2888"
server_key_iv: "964dc2d3085187d8ace5f484bd1debe1"
server_key_version: "encryptionkey@0"
I20260810 22:29:18.539736 11300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:29:18.542323 11318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:18.543567 11300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:29:18.543709 11300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "2d367e098c5f47eabc8dc2c9f45f718d"
format_stamp: "Formatted at 2026-08-10 22:29:18 on dist-test-slave-njxd"
server_key: "dcbe35041ee4fc38dcacf3787c3d2888"
server_key_iv: "964dc2d3085187d8ace5f484bd1debe1"
server_key_version: "encryptionkey@0"
I20260810 22:29:18.543825 11300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:18.557039 11300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:18.560261 11300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:18.560477 11300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:18.561074 11300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:18.561919 11300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:18.561967 11300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:18.562037 11300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:18.562081 11300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:18.570320 11300 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:45901
I20260810 22:29:18.570399 11445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:45901 every 8 connection(s)
I20260810 22:29:18.571250 11300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:29:18.579218 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 11300
I20260810 22:29:18.579373 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:29:18.579771 1910 external_mini_cluster.cc:1428] Setting key f6941f2e34ced612f686d952561702a2
Aug 10 22:29:18 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400958, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:29:18.585326 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:18.573114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:35305 (local address 127.1.221.190:35453)
0810 22:29:18.573256 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:18.573260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:18.574033 (+ 773us) server_negotiation.cc:407] Connection header received
0810 22:29:18.575318 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:18.575321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:18.575370 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:18.575451 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:18.576678 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:18.577201 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:18.578430 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:18.578657 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:18.578734 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:18.581623 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:18.581627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:18.581629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:18.581652 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:18.583062 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:18.583727 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:18.583732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:18.583736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:18.583838 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:18.584186 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:18.584190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:18.584195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:18.584466 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:18.584575 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:18.585058 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 22:29:18.585170 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:29:18.586493 11448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:18.573391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:35305)
0810 22:29:18.573876 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:18.573919 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:18.575117 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:18.575656 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:18.575662 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:18.576034 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:18.576537 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:18.576553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:18.577315 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:18.577318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:18.578312 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:18.578323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:18.578794 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:18.578796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:18.578948 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:18.579702 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:18.579726 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:18.581410 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:18.583184 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:18.583192 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:18.583210 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:18.583597 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:18.583937 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:18.583939 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:18.583940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:18.584089 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:18.584580 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:18.584585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:18.584812 (+ 227us) client_negotiation.cc:769] Sending connection context
0810 22:29:18.584971 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 22:29:18.585341 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":158,"threads_started":1}
I20260810 22:29:18.587747 11446 heartbeater.cc:347] Connected to a master server at 127.1.221.190:35453
I20260810 22:29:18.588018 11446 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:18.588675 11446 heartbeater.cc:511] Master 127.1.221.190:35453 requested a full tablet report, sending...
I20260810 22:29:18.589851 11087 ts_manager.cc:194] Registered new tserver with Master: 2d367e098c5f47eabc8dc2c9f45f718d (127.1.221.130:45901)
I20260810 22:29:18.590430 11087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:35305
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:29:18.609040 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:35453
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:18.723548 11451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:18.723839 11451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:18.723883 11451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:18.728034 11451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:18.728096 11451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:18.728173 11451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:29:18.733363 11451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:35453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:18.734436 11451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:18.735556 11451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:18.745152 11456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:18.745174 11459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:18.745174 11457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:18.745330 11451 server_base.cc:1034] running on GCE node
I20260810 22:29:18.745884 11451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:18.746433 11451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:18.747593 11451 hybrid_clock.cc:648] HybridClock initialized: now 1786400958747580 us; error 23 us; skew 500 ppm
Aug 10 22:29:18 dist-test-slave-njxd krb5kdc[11036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400958, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:18.750432 11451 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:29:18.751318 11451 webserver.cc:460] Webserver started at http://127.1.221.131:42091/ using document root <none> and password file <none>
I20260810 22:29:18.751875 11451 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:18.751937 11451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:18.752180 11451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:18.754338 11451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "ec45c8e650b2424ebef2683834e5224d"
format_stamp: "Formatted at 2026-08-10 22:29:18 on dist-test-slave-njxd"
server_key: "9fb2b37ea0cf55b1257e86b33733d6cf"
server_key_iv: "1656694e18b5490b2e0ebab1b12396ed"
server_key_version: "encryptionkey@0"
I20260810 22:29:18.754962 11451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "ec45c8e650b2424ebef2683834e5224d"
format_stamp: "Formatted at 2026-08-10 22:29:18 on dist-test-slave-njxd"
server_key: "9fb2b37ea0cf55b1257e86b33733d6cf"
server_key_iv: "1656694e18b5490b2e0ebab1b12396ed"
server_key_version: "encryptionkey@0"
I20260810 22:29:18.758783 11451 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:29:18.761298 11467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:18.762583 11451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:29:18.762705 11451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "ec45c8e650b2424ebef2683834e5224d"
format_stamp: "Formatted at 2026-08-10 22:29:18 on dist-test-slave-njxd"
server_key: "9fb2b37ea0cf55b1257e86b33733d6cf"
server_key_iv: "1656694e18b5490b2e0ebab1b12396ed"
server_key_version: "encryptionkey@0"
I20260810 22:29:18.762813 11451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:18.785122 11451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:18.788547 11451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:18.788704 11451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:18.789374 11451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:18.790261 11451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:18.790318 11451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:18.790383 11451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:18.790421 11451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:18.799675 11451 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:44569
I20260810 22:29:18.799712 11597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:44569 every 8 connection(s)
I20260810 22:29:18.800598 11451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:29:18.806259 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 11451
I20260810 22:29:18.806401 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:29:18.806712 1910 external_mini_cluster.cc:1428] Setting key b59899548ae57f9b0f54ac991d19fce5
Aug 10 22:29:18 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400958, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:29:18.814560 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:18.802452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:50207 (local address 127.1.221.190:35453)
0810 22:29:18.802603 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:18.802607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:18.803630 (+ 1023us) server_negotiation.cc:407] Connection header received
0810 22:29:18.804544 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:18.804547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:18.804609 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:18.804719 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:18.805962 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:18.806620 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:18.808056 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:18.808337 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:18.808420 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:18.811066 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:18.811069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:18.811071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:18.811096 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:18.812596 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:18.813102 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:18.813104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:18.813106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:18.813186 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:18.813497 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:18.813499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:18.813500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:18.813731 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:18.813808 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:18.814262 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 22:29:18.814430 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:29:18.815403 11601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:18.802820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:50207)
0810 22:29:18.803476 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:18.803518 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:18.804338 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:18.804905 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:18.804912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:18.805267 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:18.805815 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:18.805833 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:18.806756 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:18.806758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:18.807932 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:18.807944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:18.808492 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:18.808496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:18.808688 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:18.809300 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:18.809317 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:18.810915 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:18.812689 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:18.812693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:18.812705 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:18.813011 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:18.813267 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:18.813269 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:18.813270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:18.813416 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:18.813822 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:18.813827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:18.814056 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 22:29:18.814212 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 22:29:18.814555 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":111,"threads_started":1}
I20260810 22:29:18.816573 11598 heartbeater.cc:347] Connected to a master server at 127.1.221.190:35453
I20260810 22:29:18.816823 11598 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:18.817322 11598 heartbeater.cc:511] Master 127.1.221.190:35453 requested a full tablet report, sending...
I20260810 22:29:18.818496 11087 ts_manager.cc:194] Registered new tserver with Master: ec45c8e650b2424ebef2683834e5224d (127.1.221.131:44569)
I20260810 22:29:18.819056 11087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:50207
I20260810 22:29:18.820658 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab.
Entry for principal oryx/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab.
WARNING: no policy specified for oryx/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab.
Entry for principal oryx/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab.
WARNING: no policy specified for oryx/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab.
Entry for principal oryx/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab.
I20260810 22:29:18.898667 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 11157
I20260810 22:29:18.908157 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 11300
I20260810 22:29:18.921327 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 11451
I20260810 22:29:18.927937 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 11049
I20260810 22:29:18.936012 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:35453
--webserver_interface=127.1.221.190
--webserver_port=42695
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:35453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:19.048740 11608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:19.048990 11608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:19.049028 11608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:19.052878 11608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:19.052930 11608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:19.052949 11608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:19.052965 11608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:19.052980 11608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:29:19.058192 11608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:35453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:35453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=42695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:19.059161 11608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:19.060115 11608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:19.068365 11614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:19.068411 11617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:19.068387 11615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:19.068609 11608 server_base.cc:1034] running on GCE node
I20260810 22:29:19.069496 11608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:19.070999 11608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:19.074342 11608 hybrid_clock.cc:648] HybridClock initialized: now 1786400959074191 us; error 152 us; skew 500 ppm
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400959, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:19.078686 11608 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:29:19.080372 11608 webserver.cc:460] Webserver started at http://127.1.221.190:42695/ using document root <none> and password file <none>
I20260810 22:29:19.081175 11608 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:19.081303 11608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:19.086392 11608 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:29:19.089709 11626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:19.091432 11608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:29:19.091568 11608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "2b4a79b4181d4cf2b5afbd7291673b65"
format_stamp: "Formatted at 2026-08-10 22:29:17 on dist-test-slave-njxd"
server_key: "308548df24b4c6c3da33f50f730a41af"
server_key_iv: "c629400e8471ee9e9b4f41b7dbcfd93d"
server_key_version: "encryptionkey@0"
I20260810 22:29:19.092370 11608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:19.119279 11608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:19.122781 11608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:19.122992 11608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:19.130831 11608 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:35453
I20260810 22:29:19.130859 11686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:35453 every 8 connection(s)
I20260810 22:29:19.131927 11608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:29:19.133498 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 11608
I20260810 22:29:19.134004 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:38485
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:35453
--webserver_port=36131
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.221.129
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:29:19.136423 11687 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:29:19.138167 11687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65: Bootstrap starting.
I20260810 22:29:19.156204 11687 log.cc:824] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:19.162976 11687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:29:19.163378 11687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65: Bootstrap complete.
I20260810 22:29:19.167654 11687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35453 } }
I20260810 22:29:19.168077 11687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b4a79b4181d4cf2b5afbd7291673b65, State: Initialized, Role: FOLLOWER
I20260810 22:29:19.168570 11687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35453 } }
I20260810 22:29:19.168725 11687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:29:19.168814 11687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:29:19.168931 11687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:29:19.170950 11687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35453 } }
I20260810 22:29:19.171346 11687 leader_election.cc:302] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b4a79b4181d4cf2b5afbd7291673b65; no voters:
I20260810 22:29:19.171685 11687 leader_election.cc:288] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 22:29:19.171778 11691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:29:19.171993 11691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [term 2 LEADER]: Becoming Leader. State: Replica: 2b4a79b4181d4cf2b5afbd7291673b65, State: Running, Role: LEADER
I20260810 22:29:19.172288 11691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35453 } }
I20260810 22:29:19.172962 11687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:19.173713 11692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35453 } } }
I20260810 22:29:19.173846 11692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:19.174257 11693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b4a79b4181d4cf2b5afbd7291673b65. Latest consensus state: current_term: 2 leader_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b4a79b4181d4cf2b5afbd7291673b65" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 35453 } } }
I20260810 22:29:19.174348 11693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:19.174703 11700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:19.177980 11700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:19.178907 11700 catalog_manager.cc:1259] Loaded cluster ID: 839c663a939645349ec7889ca6a5cc72
I20260810 22:29:19.179070 11700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:19.181798 11700 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:19.182479 11700 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b4a79b4181d4cf2b5afbd7291673b65: Loaded TSK: 0
I20260810 22:29:19.183679 11700 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:29:19.255704 11689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:19.255987 11689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:19.256033 11689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:19.260212 11689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:19.260282 11689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:19.260344 11689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:29:19.265924 11689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.129.keytab
--principal=oryx/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:38485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=36131
--tserver_master_addrs=127.1.221.190:35453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:19.267023 11689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:19.268090 11689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:19.278247 11716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:19.278246 11718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:19.278244 11715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:19.278707 11689 server_base.cc:1034] running on GCE node
I20260810 22:29:19.279287 11689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:19.279946 11689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:19.281162 11689 hybrid_clock.cc:648] HybridClock initialized: now 1786400959281125 us; error 57 us; skew 500 ppm
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400959, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:19.284298 11689 init.cc:377] Logged in from keytab as oryx/127.1.221.129@KRBTEST.COM (short username oryx)
I20260810 22:29:19.285360 11689 webserver.cc:460] Webserver started at http://127.1.221.129:36131/ using document root <none> and password file <none>
I20260810 22:29:19.286026 11689 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:19.286139 11689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:19.290489 11689 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:29:19.292891 11725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:19.294296 11689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:29:19.294440 11689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "1b98ed55c17b428bbe34984570f07372"
format_stamp: "Formatted at 2026-08-10 22:29:18 on dist-test-slave-njxd"
server_key: "f3a8ff794480df7f1136e5dc07e9b13e"
server_key_iv: "f6baa9bc045ae7cfd9f2655915520e8e"
server_key_version: "encryptionkey@0"
I20260810 22:29:19.295030 11689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:19.307693 11689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:19.310988 11689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:19.311199 11689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:19.311920 11689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:19.312783 11689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:19.312865 11689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:19.312934 11689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:19.312966 11689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:19.323000 11689 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:38485
I20260810 22:29:19.323021 11853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:38485 every 8 connection(s)
I20260810 22:29:19.324048 11689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:29:19.331996 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 11689
I20260810 22:29:19.332548 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:45901
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:35453
--webserver_port=33365
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.221.130
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:19.337107 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:19.326351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:55299)
0810 22:29:19.327065 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:19.327107 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:19.327880 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:19.328923 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:19.328929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:19.329285 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:19.329783 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.329797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.331040 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.331042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.331910 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.331919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.332995 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.332998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.333139 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.333820 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:19.333837 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:19.336317 (+ 2480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":143,"threads_started":1}
W20260810 22:29:19.337661 11856 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:19.326191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:55299 (local address 127.1.221.190:35453)
0810 22:29:19.326670 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:19.326692 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:19.327228 (+ 536us) server_negotiation.cc:407] Connection header received
0810 22:29:19.328235 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:19.328246 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:19.328544 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:19.328829 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:19.329939 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.330914 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.332040 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.332831 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.333020 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.336635 (+ 3615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:55299: BlockingRecv error: recv got EOF from 127.1.221.129:55299 (error 108)
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":121,"threads_started":1}
W20260810 22:29:19.337713 11854 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:35453 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.221.190:35453: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:19.343851 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:19.338156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:58829)
0810 22:29:19.338331 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:19.338345 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:19.338430 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:19.338778 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:19.338780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:19.338916 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:19.339008 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.339013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.340144 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.340147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.340681 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.340688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.341366 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.341369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.341436 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.341959 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:19.341971 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:19.343744 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20260810 22:29:19.344122 11856 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:19.338370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:58829 (local address 127.1.221.190:35453)
0810 22:29:19.338506 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:19.338512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:19.338529 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:29:19.338578 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:19.338584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:19.338646 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:19.338750 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:19.339130 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.340028 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.340813 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.341239 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.341376 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.343962 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:58829: BlockingRecv error: recv got EOF from 127.1.221.129:58829 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:19.349433 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:19.344382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:45061)
0810 22:29:19.344575 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:19.344591 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:19.344682 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:19.344934 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:19.344937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:19.345090 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:19.345160 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.345165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.345961 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.345965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.346480 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.346487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.346931 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.346935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.347021 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.347504 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:19.347516 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:19.349308 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126}
W20260810 22:29:19.349566 11856 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:19.344459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:45061 (local address 127.1.221.190:35453)
0810 22:29:19.344594 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:19.344599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:19.344634 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:29:19.344784 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:19.344793 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:19.344833 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:19.344915 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:19.345287 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.345852 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.346604 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.346817 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.346920 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.349433 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:45061: BlockingRecv error: recv got EOF from 127.1.221.129:45061 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 22:29:19.349875 11854 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:29:19.450690 11858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:19.450964 11858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:19.451025 11858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:19.455385 11858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:19.455447 11858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:19.455539 11858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:29:19.460731 11858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.130.keytab
--principal=oryx/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:45901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=33365
--tserver_master_addrs=127.1.221.190:35453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:19.461867 11858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:19.462927 11858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:19.473012 11867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:19.473012 11864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:19.473106 11865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:19.473531 11858 server_base.cc:1034] running on GCE node
I20260810 22:29:19.474026 11858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:19.474854 11858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:19.476016 11858 hybrid_clock.cc:648] HybridClock initialized: now 1786400959475979 us; error 48 us; skew 500 ppm
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400959, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:19.479012 11858 init.cc:377] Logged in from keytab as oryx/127.1.221.130@KRBTEST.COM (short username oryx)
I20260810 22:29:19.479844 11858 webserver.cc:460] Webserver started at http://127.1.221.130:33365/ using document root <none> and password file <none>
I20260810 22:29:19.480418 11858 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:19.480520 11858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:19.484434 11858 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260810 22:29:19.486563 11874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:19.487794 11858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:29:19.487931 11858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "2d367e098c5f47eabc8dc2c9f45f718d"
format_stamp: "Formatted at 2026-08-10 22:29:18 on dist-test-slave-njxd"
server_key: "dcbe35041ee4fc38dcacf3787c3d2888"
server_key_iv: "964dc2d3085187d8ace5f484bd1debe1"
server_key_version: "encryptionkey@0"
I20260810 22:29:19.488464 11858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:19.508795 11858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:19.512108 11858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:19.512326 11858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:19.512933 11858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:19.513835 11858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:19.513904 11858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:19.514003 11858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:19.514053 11858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:19.523110 11858 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:45901
I20260810 22:29:19.523137 11995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:45901 every 8 connection(s)
I20260810 22:29:19.524379 11858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:29:19.530799 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 11858
I20260810 22:29:19.531239 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:44569
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:35453
--webserver_port=42091
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.221.131
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 22:29:19.535964 11856 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:19.526382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:38813 (local address 127.1.221.190:35453)
0810 22:29:19.526536 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:19.526541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:19.527644 (+ 1103us) server_negotiation.cc:407] Connection header received
0810 22:29:19.528466 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:19.528469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:19.528521 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:19.528596 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:19.529794 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.530313 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.531606 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.531826 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.531934 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.535866 (+ 3932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:38813: BlockingRecv error: recv got EOF from 127.1.221.130:38813 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:29:19.536855 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:19.526842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:38813)
0810 22:29:19.527496 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:19.527538 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:19.528291 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:19.528769 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:19.528778 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:19.529125 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:19.529654 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.529678 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.530420 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.530423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.531471 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.531480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.531976 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.531980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.532147 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.532769 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:19.532786 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:19.535734 (+ 2948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":106,"threads_started":1}
W20260810 22:29:19.537319 11996 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:35453 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.221.190:35453: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:19.543102 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:19.537799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:57701)
0810 22:29:19.537894 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:19.537908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:19.537978 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:19.538295 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:19.538296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:19.538418 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:19.538499 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.538503 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.539550 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.539552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.540055 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.540062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.540676 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.540678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.540737 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.541221 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:19.541233 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:19.543017 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 22:29:19.543375 11856 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:19.537902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:57701 (local address 127.1.221.190:35453)
0810 22:29:19.538042 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:19.538047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:19.538063 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:29:19.538109 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:19.538114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:19.538167 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:19.538262 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:19.538611 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.539443 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.540167 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.540568 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.540687 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.543211 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:57701: BlockingRecv error: recv got EOF from 127.1.221.130:57701 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:19.548606 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:19.543696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:59757)
0810 22:29:19.543788 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:19.543802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:19.543882 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:19.544129 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:19.544131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:19.544245 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:19.544316 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.544321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.545251 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.545253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.545826 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.545834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.546355 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.546357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.546479 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.546922 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:19.546933 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:19.548529 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 22:29:19.548756 11856 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:19.543784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:59757 (local address 127.1.221.190:35453)
0810 22:29:19.543909 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:19.543915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:19.543932 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:29:19.543974 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:19.543979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:19.544026 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:19.544108 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:19.544416 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.545150 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.545926 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.546246 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.546350 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.548655 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:59757: BlockingRecv error: recv got EOF from 127.1.221.130:59757 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 22:29:19.548800 11996 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:29:19.648015 11999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:19.648339 11999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:19.648429 11999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:19.652688 11999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:19.652844 11999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:19.653000 11999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:29:19.658931 11999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/krb5kdc/oryx_127.1.221.131.keytab
--principal=oryx/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:44569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=42091
--tserver_master_addrs=127.1.221.190:35453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:19.660000 11999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:19.661024 11999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:19.671013 12007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:19.671034 12006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:19.671013 12009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:19.671336 11999 server_base.cc:1034] running on GCE node
I20260810 22:29:19.671883 11999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:19.672549 11999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:19.673707 11999 hybrid_clock.cc:648] HybridClock initialized: now 1786400959673687 us; error 29 us; skew 500 ppm
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400959, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:19.676754 11999 init.cc:377] Logged in from keytab as oryx/127.1.221.131@KRBTEST.COM (short username oryx)
I20260810 22:29:19.677721 11999 webserver.cc:460] Webserver started at http://127.1.221.131:42091/ using document root <none> and password file <none>
I20260810 22:29:19.678306 11999 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:19.678426 11999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:19.682469 11999 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 22:29:19.684563 12017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:19.685829 11999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:29:19.685961 11999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "ec45c8e650b2424ebef2683834e5224d"
format_stamp: "Formatted at 2026-08-10 22:29:18 on dist-test-slave-njxd"
server_key: "9fb2b37ea0cf55b1257e86b33733d6cf"
server_key_iv: "1656694e18b5490b2e0ebab1b12396ed"
server_key_version: "encryptionkey@0"
I20260810 22:29:19.686509 11999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:19.723018 11999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:19.726657 11999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:19.726900 11999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:19.727571 11999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:19.728458 11999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:19.728534 11999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:19.728612 11999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:19.728664 11999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:19.738209 11999 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:44569
I20260810 22:29:19.738281 12142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:44569 every 8 connection(s)
I20260810 22:29:19.739321 11999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:29:19.740137 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 11999
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:19.750511 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:19.740958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.1.221.190:35453)
0810 22:29:19.741123 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:19.741128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:19.741463 (+ 335us) server_negotiation.cc:407] Connection header received
0810 22:29:19.741648 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:19.741652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:19.741708 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:19.741806 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:19.742604 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.743139 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.744121 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.744368 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.744482 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.746073 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:19.746079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:19.746094 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:19.746125 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:19.747936 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:19.748454 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:19.748461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:19.748469 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:19.748616 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:19.748929 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:19.748933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:19.748935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:19.749435 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:19.749782 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:19.750003 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 22:29:19.750303 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 22:29:19.751587 12147 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:19.742760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:52487 (local address 127.1.221.190:35453)
0810 22:29:19.743096 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:19.743103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:19.743120 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:29:19.744007 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:19.744010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:19.744062 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:19.744150 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:19.745325 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.745908 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.747443 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.747679 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.747806 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.751475 (+ 3669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:52487: BlockingRecv error: recv got EOF from 127.1.221.131:52487 (error 108)
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":90,"threads_started":1}
I20260810 22:29:19.752265 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:19.741692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:52487)
0810 22:29:19.742237 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:19.742301 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:19.743799 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:19.744333 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:19.744341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:19.744690 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:19.745169 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.745182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.746053 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.746057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.747300 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.747314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.747853 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.747858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.748050 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.748632 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:19.748648 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:19.751372 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":167,"threads_started":1}
W20260810 22:29:19.752877 12143 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:35453 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.221.190:35453: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:19.758245 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:19.753346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:46353)
0810 22:29:19.753515 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:19.753528 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:19.753631 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:19.753857 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:19.753859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:19.754020 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:19.754106 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.754111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.754838 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.754843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.755385 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.755392 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.755850 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.755854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.755946 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.756420 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:19.756440 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:19.758159 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260810 22:29:19.758423 12147 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:19.753424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:46353 (local address 127.1.221.190:35453)
0810 22:29:19.753559 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:19.753565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:19.753600 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:29:19.753709 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:19.753712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:19.753756 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:19.753842 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:19.754218 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.754729 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.755505 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.755722 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.755850 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.758306 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:46353: BlockingRecv error: recv got EOF from 127.1.221.131:46353 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:19 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:19.762929 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:19.758826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:55881)
0810 22:29:19.758917 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:19.758932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:19.759002 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:19.759157 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:19.759159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:19.759274 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:19.759344 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.759348 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.759974 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.759976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.760471 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:19.760478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.760855 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.760857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:19.760911 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.761268 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:19.761278 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:19.762848 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 22:29:19.763070 12147 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:19.758869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:55881 (local address 127.1.221.190:35453)
0810 22:29:19.759000 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:19.759009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:19.759021 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:29:19.759053 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:19.759055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:19.759087 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:19.759144 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:19.759431 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.759885 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.760563 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:19.760755 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:19.760833 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:19.762967 (+ 2134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:55881: BlockingRecv error: recv got EOF from 127.1.221.131:55881 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 22:29:19.763231 12143 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 22:29:20 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:20 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:20.356077 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:20.350641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:55473)
0810 22:29:20.350952 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:20.350969 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:20.351094 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:20.351438 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:20.351441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:20.351630 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:20.351753 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:20.351758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:20.352490 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:20.352493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:20.353060 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:20.353066 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:20.353482 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:20.353484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:20.353558 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:20.354067 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:20.354078 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:20.355962 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":104,"threads_started":1}
W20260810 22:29:20.356232 12149 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:20.350703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:55473 (local address 127.1.221.190:35453)
0810 22:29:20.350973 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:20.350978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:20.351027 (+ 49us) server_negotiation.cc:407] Connection header received
0810 22:29:20.351229 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:20.351232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:20.351299 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:20.351390 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:20.351860 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:20.352393 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:20.353157 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:20.353382 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:20.353456 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:20.356097 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:55473: BlockingRecv error: recv got EOF from 127.1.221.129:55473 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
Aug 10 22:29:20 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:20 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:20.554930 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:20.549544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:47983)
0810 22:29:20.549838 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:20.549854 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:20.549996 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:20.550315 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:20.550319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:20.550518 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:20.550650 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:20.550656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:20.551373 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:20.551375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:20.551949 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:20.551957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:20.552362 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:20.552365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:20.552439 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:20.552909 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:20.552920 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:20.554810 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
W20260810 22:29:20.555075 12149 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:20.549623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:47983 (local address 127.1.221.190:35453)
0810 22:29:20.549804 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:20.549810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:20.549943 (+ 133us) server_negotiation.cc:407] Connection header received
0810 22:29:20.550089 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:20.550092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:20.550172 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:20.550284 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:20.550755 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:20.551275 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:20.552047 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:20.552262 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:20.552337 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:20.554955 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:47983: BlockingRecv error: recv got EOF from 127.1.221.130:47983 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 22:29:20 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:20 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:20.768939 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:20.763812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:46355)
0810 22:29:20.764060 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:20.764073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:20.764170 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:20.764393 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:20.764395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:20.764575 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:20.764672 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:20.764677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:20.765378 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:20.765380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:20.766001 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:20.766010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:20.766426 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:20.766429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:20.766490 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:20.766938 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:20.766950 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:20.768803 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
W20260810 22:29:20.769052 12149 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:20.763839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:46355 (local address 127.1.221.190:35453)
0810 22:29:20.763983 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:20.763988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:20.764144 (+ 156us) server_negotiation.cc:407] Connection header received
0810 22:29:20.764237 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:20.764239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:20.764291 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:20.764358 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:20.764766 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:20.765284 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:20.766102 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:20.766325 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:20.766407 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:20.768937 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:46355: BlockingRecv error: recv got EOF from 127.1.221.131:46355 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 22:29:21 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:21 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:21.362890 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:21.357175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:55683)
0810 22:29:21.357538 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:21.357557 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:21.357721 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:21.358069 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:21.358072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:21.358312 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:21.358425 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:21.358431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:21.359278 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:21.359280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:21.359855 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:21.359862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:21.360298 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:21.360301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:21.360362 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:21.360865 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:21.360877 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:21.362763 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":88,"threads_started":1}
W20260810 22:29:21.363108 12154 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:21.357240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:55683 (local address 127.1.221.190:35453)
0810 22:29:21.357588 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:21.357593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:21.357661 (+ 68us) server_negotiation.cc:407] Connection header received
0810 22:29:21.357861 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:21.357864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:21.357924 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:21.358022 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:21.358602 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:21.359165 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:21.359967 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:21.360196 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:21.360278 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:21.362933 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:55683: BlockingRecv error: recv got EOF from 127.1.221.129:55683 (error 108)
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
Aug 10 22:29:21 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:21 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:21.561692 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:21.555873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:51541)
0810 22:29:21.556204 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:21.556221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:21.556350 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:21.556650 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:21.556653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:21.556883 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:21.556985 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:21.556990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:21.557857 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:21.557862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:21.558440 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:21.558448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:21.558947 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:21.558951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:21.559042 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:21.559639 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:21.559651 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:21.561482 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":84,"threads_started":1}
W20260810 22:29:21.561774 12154 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:21.555930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:51541 (local address 127.1.221.190:35453)
0810 22:29:21.556101 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:21.556106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:21.556285 (+ 179us) server_negotiation.cc:407] Connection header received
0810 22:29:21.556467 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:21.556472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:21.556522 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:21.556618 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:21.557115 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:21.557740 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:21.558570 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:21.558823 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:21.558931 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:21.561658 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:51541: BlockingRecv error: recv got EOF from 127.1.221.130:51541 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 22:29:21 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:21 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:21.775557 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:21.769966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:55665)
0810 22:29:21.770292 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:21.770306 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:21.770407 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:21.770653 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:21.770655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:21.770841 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:21.770941 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:21.770946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:21.771671 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:21.771673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:21.772214 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:21.772221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:21.772637 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:21.772639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:21.772700 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:21.773188 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:21.773199 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:21.775411 (+ 2212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":133,"threads_started":1}
W20260810 22:29:21.775635 12154 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:21.770107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:55665 (local address 127.1.221.190:35453)
0810 22:29:21.770273 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:21.770276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:21.770374 (+ 98us) server_negotiation.cc:407] Connection header received
0810 22:29:21.770490 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:21.770493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:21.770539 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:21.770612 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:21.771043 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:21.771569 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:21.772329 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:21.772538 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:21.772613 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:21.775537 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:55665: BlockingRecv error: recv got EOF from 127.1.221.131:55665 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 22:29:22 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:22 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:22.369675 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:22.363844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:33585)
0810 22:29:22.364162 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:22.364180 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:22.364287 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:22.364698 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:22.364701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:22.364932 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:22.365037 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:22.365043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:22.365923 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:22.365926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:22.366494 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:22.366502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:22.366928 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:22.366930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:22.366994 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:22.367590 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:22.367603 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:22.369476 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
W20260810 22:29:22.369752 12158 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:22.363961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:33585 (local address 127.1.221.190:35453)
0810 22:29:22.364408 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:22.364412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:22.364424 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:29:22.364486 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:22.364489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:22.364540 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:22.364638 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:22.365163 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:22.365821 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:22.366607 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:22.366821 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:22.366897 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:22.369645 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:33585: BlockingRecv error: recv got EOF from 127.1.221.129:33585 (error 108)
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":110,"threads_started":1}
Aug 10 22:29:22 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:22 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:22.568604 12159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:22.562813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:36467)
0810 22:29:22.563151 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:22.563170 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:22.563315 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:22.563600 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:22.563602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:22.563835 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:22.563946 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:22.563951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:22.564735 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:22.564741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:22.565381 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:22.565389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:22.565913 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:22.565916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:22.566003 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:22.566493 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:22.566504 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:22.568417 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":113,"threads_started":1}
W20260810 22:29:22.568677 12158 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:22.562926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:36467 (local address 127.1.221.190:35453)
0810 22:29:22.563105 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:22.563110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:22.563248 (+ 138us) server_negotiation.cc:407] Connection header received
0810 22:29:22.563420 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:22.563423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:22.563479 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:22.563572 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:22.564103 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:22.564619 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:22.565509 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:22.565793 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:22.565906 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:22.568560 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:36467: BlockingRecv error: recv got EOF from 127.1.221.130:36467 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 10 22:29:22 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:22 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:22.782042 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:22.776549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:57163)
0810 22:29:22.776855 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:22.776867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:22.776993 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:22.777287 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:22.777290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:22.777506 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:22.777632 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:22.777638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:22.778422 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:22.778424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:22.778973 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:22.778981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:22.779387 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:22.779389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:22.779450 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:22.780035 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:22.780048 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:22.781929 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":94,"threads_started":1}
W20260810 22:29:22.782191 12158 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:22.776611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:57163 (local address 127.1.221.190:35453)
0810 22:29:22.776786 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:22.776793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:22.776941 (+ 148us) server_negotiation.cc:407] Connection header received
0810 22:29:22.777102 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:22.777106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:22.777165 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:22.777253 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:22.777781 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:22.778318 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:22.779078 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:22.779289 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:22.779364 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:22.782068 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:57163: BlockingRecv error: recv got EOF from 127.1.221.131:57163 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 22:29:23 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:23 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:23.381253 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:23.370704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:58587)
0810 22:29:23.371014 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:23.371029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:23.371125 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:23.371377 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:23.371379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:23.371563 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:23.371663 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:23.371668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:23.372398 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:23.372400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:23.372953 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:23.372961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:23.373648 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:23.373652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:23.373755 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:23.379110 (+ 5355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:23.379126 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:23.381079 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":122,"threads_started":1}
W20260810 22:29:23.381348 12162 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:23.370817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:58587 (local address 127.1.221.190:35453)
0810 22:29:23.371060 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:23.371064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:23.371080 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:29:23.371218 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:23.371220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:23.371263 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:23.371337 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:23.371779 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:23.372302 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:23.373055 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:23.373278 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:23.373433 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:23.381218 (+ 7785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:58587: BlockingRecv error: recv got EOF from 127.1.221.129:58587 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":89,"threads_started":1}
Aug 10 22:29:23 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:23 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:23.575686 12164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:23.569552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:60497)
0810 22:29:23.569904 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:23.569918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:23.570067 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:23.570379 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:23.570382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:23.570606 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:23.570705 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:23.570711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:23.571503 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:23.571508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:23.572083 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:23.572091 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:23.572579 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:23.572583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:23.572672 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:23.573242 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:23.573256 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:23.575570 (+ 2314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":89,"threads_started":1}
W20260810 22:29:23.575842 12162 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:23.569658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:60497 (local address 127.1.221.190:35453)
0810 22:29:23.569829 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:23.569834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:23.570010 (+ 176us) server_negotiation.cc:407] Connection header received
0810 22:29:23.570194 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:23.570197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:23.570251 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:23.570359 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:23.570836 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:23.571380 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:23.572211 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:23.572457 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:23.572572 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:23.575713 (+ 3141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:60497: BlockingRecv error: recv got EOF from 127.1.221.130:60497 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 22:29:23 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:23 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:23.788915 12165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:23.782886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:37137)
0810 22:29:23.783249 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:23.783264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:23.783405 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:23.783704 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:23.783706 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:23.783934 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:23.784031 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:23.784036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:23.784819 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:23.784823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:23.785658 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:23.785673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:23.786302 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:23.786305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:23.786392 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:23.787042 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:23.787055 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:23.788799 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
W20260810 22:29:23.789112 12162 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:23.783014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:37137 (local address 127.1.221.190:35453)
0810 22:29:23.783201 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:23.783206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:23.783348 (+ 142us) server_negotiation.cc:407] Connection header received
0810 22:29:23.783525 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:23.783528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:23.783585 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:23.783676 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:23.784153 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:23.784697 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:23.785808 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:23.786037 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:23.786278 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:23.788981 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:37137: BlockingRecv error: recv got EOF from 127.1.221.131:37137 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 10 22:29:24 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:24 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:24.387948 12166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:24.382297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:54749)
0810 22:29:24.382652 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:24.382671 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:24.382792 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:24.383106 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:24.383109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:24.383356 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:24.383454 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:24.383460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:24.384221 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:24.384223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:24.384817 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:24.384838 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:24.385261 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:24.385264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:24.385333 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:24.385880 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:24.385892 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:24.387775 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":90,"threads_started":1}
W20260810 22:29:24.388048 12167 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:24.382400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:54749 (local address 127.1.221.190:35453)
0810 22:29:24.382687 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:24.382692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:24.382731 (+ 39us) server_negotiation.cc:407] Connection header received
0810 22:29:24.382897 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:24.382902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:24.382958 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:24.383048 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:24.383580 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:24.384129 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:24.384932 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:24.385143 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:24.385236 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:24.387914 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:54749: BlockingRecv error: recv got EOF from 127.1.221.129:54749 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
Aug 10 22:29:24 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:24 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:24.582314 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:24.576666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:58877)
0810 22:29:24.576988 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:24.577001 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:24.577088 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:24.577339 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:24.577341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:24.577535 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:24.577652 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:24.577657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:24.578368 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:24.578370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:24.579177 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:24.579188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:24.579675 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:24.579680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:24.579764 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:24.580349 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:24.580361 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:24.582186 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":124,"threads_started":1}
W20260810 22:29:24.582448 12167 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:24.576888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:58877 (local address 127.1.221.190:35453)
0810 22:29:24.577045 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:24.577048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:24.577065 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:29:24.577172 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:24.577174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:24.577219 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:24.577301 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:24.577749 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:24.578252 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:24.579313 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:24.579566 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:24.579648 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:24.582334 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:58877: BlockingRecv error: recv got EOF from 127.1.221.130:58877 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 22:29:24 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:24 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:24.795825 12169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:24.789967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:33645)
0810 22:29:24.790234 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:24.790257 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:24.790383 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:24.790685 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:24.790691 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:24.790940 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:24.791050 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:24.791056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:24.791976 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:24.791978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:24.792661 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:24.792670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:24.793093 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:24.793095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:24.793156 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:24.793704 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:24.793716 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:24.795652 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
W20260810 22:29:24.795892 12167 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:24.790053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:33645 (local address 127.1.221.190:35453)
0810 22:29:24.790209 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:24.790214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:24.790336 (+ 122us) server_negotiation.cc:407] Connection header received
0810 22:29:24.790474 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:24.790476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:24.790549 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:24.790634 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:24.791143 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:24.791844 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:24.792763 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:24.792991 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:24.793074 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:24.795773 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:33645: BlockingRecv error: recv got EOF from 127.1.221.131:33645 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 22:29:25 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:25 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:25.394253 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:25.388769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:38011)
0810 22:29:25.389061 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:25.389080 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:25.389172 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:25.389453 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:25.389456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:25.389681 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:25.389784 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:25.389790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:25.390558 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:25.390561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:25.391114 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:25.391122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:25.391551 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:25.391553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:25.391616 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:25.392159 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:25.392171 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:25.394138 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":97,"threads_started":1}
W20260810 22:29:25.394462 12171 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:25.388852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:38011 (local address 127.1.221.190:35453)
0810 22:29:25.389126 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:25.389130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:25.389144 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:29:25.389255 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:25.389257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:25.389309 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:25.389412 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:25.389892 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:25.390424 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:25.391218 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:25.391451 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:25.391530 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:25.394305 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:38011: BlockingRecv error: recv got EOF from 127.1.221.129:38011 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
Aug 10 22:29:25 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:25 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:25.589195 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:25.583361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:56445)
0810 22:29:25.583685 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:25.583698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:25.583857 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:25.584149 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:25.584152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:25.584387 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:25.584487 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:25.584493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:25.585278 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:25.585284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:25.585901 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:25.585910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:25.586394 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:25.586398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:25.586492 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:25.587067 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:25.587079 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:25.589016 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":94,"threads_started":1}
W20260810 22:29:25.589263 12171 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:25.583440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:56445 (local address 127.1.221.190:35453)
0810 22:29:25.583626 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:25.583631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:25.583782 (+ 151us) server_negotiation.cc:407] Connection header received
0810 22:29:25.583965 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:25.583969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:25.584025 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:25.584121 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:25.584626 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:25.585162 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:25.586027 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:25.586258 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:25.586378 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:25.589152 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:56445: BlockingRecv error: recv got EOF from 127.1.221.130:56445 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 10 22:29:25 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:25 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:25.802649 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:25.796671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:38189)
0810 22:29:25.797009 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:25.797024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:25.797129 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:25.797432 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:25.797434 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:25.797695 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:25.797819 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:25.797826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:25.798884 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:25.798887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:25.799668 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:25.799679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:25.800093 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:25.800095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:25.800157 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:25.800641 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:25.800652 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:25.802531 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":160,"threads_started":1}
W20260810 22:29:25.802794 12171 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:25.796800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:38189 (local address 127.1.221.190:35453)
0810 22:29:25.796924 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:25.796927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:25.797095 (+ 168us) server_negotiation.cc:407] Connection header received
0810 22:29:25.797253 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:25.797255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:25.797306 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:25.797372 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:25.797991 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:25.798767 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:25.799776 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:25.799992 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:25.800073 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:25.802671 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:38189: BlockingRecv error: recv got EOF from 127.1.221.131:38189 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 10 22:29:26 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:26 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:26.400842 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:26.395080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:44331)
0810 22:29:26.395454 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:26.395470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:26.395573 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:26.395859 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:26.395861 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:26.396054 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:26.396151 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:26.396157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:26.396910 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:26.396913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:26.397490 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:26.397498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:26.398068 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:26.398071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:26.398176 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:26.398778 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:26.398799 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:26.400721 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":101,"threads_started":1}
W20260810 22:29:26.401031 12175 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:26.395175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:44331 (local address 127.1.221.190:35453)
0810 22:29:26.395452 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:26.395456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:26.395545 (+ 89us) server_negotiation.cc:407] Connection header received
0810 22:29:26.395654 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:26.395656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:26.395705 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:26.395786 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:26.396262 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:26.396809 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:26.397646 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:26.397938 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:26.398073 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:26.400860 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:44331: BlockingRecv error: recv got EOF from 127.1.221.129:44331 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
Aug 10 22:29:26 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:26 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:26.595945 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:26.590076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:41375)
0810 22:29:26.590527 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:26.590541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:26.590643 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:26.590900 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:26.590903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:26.591099 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:26.591198 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:26.591203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:26.592031 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:26.592034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:26.592631 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:26.592639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:26.593172 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:26.593175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:26.593240 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:26.593871 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:26.593884 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:26.595824 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":136,"threads_started":1}
W20260810 22:29:26.596174 12175 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:26.590134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:41375 (local address 127.1.221.190:35453)
0810 22:29:26.590284 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:26.590288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:26.590619 (+ 331us) server_negotiation.cc:407] Connection header received
0810 22:29:26.590725 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:26.590727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:26.590779 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:26.590867 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:26.591302 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:26.591881 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:26.592767 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:26.592999 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:26.593160 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:26.596029 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:41375: BlockingRecv error: recv got EOF from 127.1.221.130:41375 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 22:29:26 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:26 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:26.809144 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:26.803465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:54933)
0810 22:29:26.803845 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:26.803858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:26.803994 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:26.804280 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:26.804283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:26.804523 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:26.804613 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:26.804619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:26.805383 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:26.805387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:26.805967 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:26.805976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:26.806455 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:26.806458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:26.806548 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:26.807117 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:26.807130 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:26.808975 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":101,"threads_started":1}
W20260810 22:29:26.809203 12175 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:26.803530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:54933 (local address 127.1.221.190:35453)
0810 22:29:26.803691 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:26.803698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:26.803940 (+ 242us) server_negotiation.cc:407] Connection header received
0810 22:29:26.804103 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:26.804107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:26.804163 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:26.804252 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:26.804734 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:26.805243 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:26.806087 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:26.806336 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:26.806448 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:26.809097 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:54933: BlockingRecv error: recv got EOF from 127.1.221.131:54933 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 22:29:27 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:27 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:27.406930 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:27.401628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:47933)
0810 22:29:27.401932 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:27.401946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:27.402035 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:27.402292 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:27.402294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:27.402482 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:27.402583 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:27.402588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:27.403339 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:27.403342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:27.403886 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:27.403893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:27.404312 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:27.404315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:27.404376 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:27.404878 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:27.404889 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:27.406814 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":119,"threads_started":1}
W20260810 22:29:27.407096 12180 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:27.401766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:47933 (local address 127.1.221.190:35453)
0810 22:29:27.402067 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:27.402070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:27.402084 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:29:27.402121 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:27.402123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:27.402165 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:27.402246 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:27.402700 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:27.403236 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:27.403989 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:27.404208 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:27.404294 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:27.406958 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:47933: BlockingRecv error: recv got EOF from 127.1.221.129:47933 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":127,"threads_started":1}
Aug 10 22:29:27 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:27 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:27.602684 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:27.596775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:57207)
0810 22:29:27.597116 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:27.597130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:27.597242 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:27.597558 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:27.597561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:27.597810 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:27.597960 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:27.597967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:27.598950 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:27.598954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:27.599566 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:27.599577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:27.600047 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:27.600050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:27.600158 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:27.600642 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:27.600654 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:27.602563 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":90,"threads_started":1}
W20260810 22:29:27.602900 12180 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:27.596958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:57207 (local address 127.1.221.190:35453)
0810 22:29:27.597132 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:27.597136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:27.597218 (+ 82us) server_negotiation.cc:407] Connection header received
0810 22:29:27.597374 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:27.597378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:27.597432 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:27.597526 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:27.598097 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:27.598833 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:27.599692 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:27.599921 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:27.600029 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:27.602766 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:57207: BlockingRecv error: recv got EOF from 127.1.221.130:57207 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 22:29:27 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:27 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:27.815354 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:27.809964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:51201)
0810 22:29:27.810297 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:27.810312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:27.810413 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:27.810655 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:27.810657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:27.810857 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:27.810950 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:27.810955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:27.811718 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:27.811720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:27.812256 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:27.812263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:27.812754 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:27.812757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:27.812821 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:27.813321 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:27.813333 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:27.815232 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":131,"threads_started":1}
W20260810 22:29:27.815469 12180 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:27.810016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:51201 (local address 127.1.221.190:35453)
0810 22:29:27.810168 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:27.810172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:27.810385 (+ 213us) server_negotiation.cc:407] Connection header received
0810 22:29:27.810494 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:27.810496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:27.810544 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:27.810637 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:27.811070 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:27.811622 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:27.812405 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:27.812637 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:27.812730 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:27.815374 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:51201: BlockingRecv error: recv got EOF from 127.1.221.131:51201 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 22:29:28 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:28 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:28.416718 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:28.407892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:59531)
0810 22:29:28.408183 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:28.408198 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:28.408301 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:28.408619 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:28.408622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:28.408809 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:28.408907 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:28.408912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:28.409731 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:28.409734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:28.410299 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:28.410307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:28.410742 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:28.410744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:28.410807 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:28.414725 (+ 3918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:28.414738 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:28.416602 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
W20260810 22:29:28.416877 12185 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:28.408112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:59531 (local address 127.1.221.190:35453)
0810 22:29:28.408390 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:28.408393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:28.408407 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:29:28.408447 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:28.408449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:28.408491 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:28.408576 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:28.409017 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:28.409557 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:28.410403 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:28.410629 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:28.410719 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:28.416741 (+ 6022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:59531: BlockingRecv error: recv got EOF from 127.1.221.129:59531 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":107,"threads_started":1}
Aug 10 22:29:28 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:28 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:28.608683 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:28.603428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:51771)
0810 22:29:28.603729 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:28.603742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:28.603844 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:28.604088 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:28.604090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:28.604289 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:28.604392 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:28.604397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:28.605115 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:28.605117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:28.605717 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:28.605726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:28.606142 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:28.606144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:28.606208 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:28.606697 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:28.606709 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:28.608563 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":122,"threads_started":1}
W20260810 22:29:28.608832 12185 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:28.603538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:51771 (local address 127.1.221.190:35453)
0810 22:29:28.603681 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:28.603685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:28.603811 (+ 126us) server_negotiation.cc:407] Connection header received
0810 22:29:28.603926 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:28.603928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:28.603974 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:28.604044 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:28.604495 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:28.605017 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:28.605816 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:28.606037 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:28.606114 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:28.608712 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:51771: BlockingRecv error: recv got EOF from 127.1.221.130:51771 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 22:29:28 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:28 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:28.822247 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:28.816417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:43213)
0810 22:29:28.816692 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:28.816706 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:28.816809 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:28.817073 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:28.817075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:28.817263 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:28.817368 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:28.817374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:28.818351 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:28.818355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:28.819210 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:28.819219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:28.819729 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:28.819732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:28.819800 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:28.820297 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:28.820308 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:28.822111 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
W20260810 22:29:28.822377 12185 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:28.816568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:43213 (local address 127.1.221.190:35453)
0810 22:29:28.816758 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:28.816762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:28.816778 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:29:28.816906 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:28.816908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:28.816957 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:28.817028 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:28.817471 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:28.818233 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:28.819364 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:28.819618 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:28.819706 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:28.822242 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:43213: BlockingRecv error: recv got EOF from 127.1.221.131:43213 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 10 22:29:29 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:29 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:29.423242 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:29.417746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:36079)
0810 22:29:29.418044 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:29.418063 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:29.418166 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:29.418430 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:29.418433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:29.418654 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:29.418750 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:29.418755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:29.419557 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:29.419564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:29.420142 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:29.420149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:29.420643 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:29.420647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:29.420733 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:29.421234 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:29.421246 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:29.423075 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":72,"threads_started":1}
W20260810 22:29:29.423349 12189 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:29.417890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:36079 (local address 127.1.221.190:35453)
0810 22:29:29.418177 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:29.418181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:29.418216 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:29:29.418253 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:29.418255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:29.418298 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:29.418365 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:29.418887 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:29.419429 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:29.420267 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:29.420509 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:29.420636 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:29.423215 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:36079: BlockingRecv error: recv got EOF from 127.1.221.129:36079 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":104,"threads_started":1}
Aug 10 22:29:29 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:29 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:29.620013 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:29.609731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:43129)
0810 22:29:29.610104 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:29.610117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:29.610254 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:29.610549 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:29.610552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:29.610773 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:29.610872 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:29.610877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:29.611654 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:29.611659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:29.612230 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:29.612237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:29.612727 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:29.612731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:29.612820 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:29.617821 (+ 5001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:29.617833 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:29.619820 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":120,"threads_started":1}
W20260810 22:29:29.620095 12189 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:29.609862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:43129 (local address 127.1.221.190:35453)
0810 22:29:29.610050 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:29.610055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:29.610202 (+ 147us) server_negotiation.cc:407] Connection header received
0810 22:29:29.610375 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:29.610378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:29.610428 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:29.610524 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:29.611011 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:29.611544 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:29.612365 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:29.612610 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:29.612722 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:29.619961 (+ 7239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:43129: BlockingRecv error: recv got EOF from 127.1.221.130:43129 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 10 22:29:29 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:29 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:29.829319 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:29.823265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:34465)
0810 22:29:29.823633 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:29.823650 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:29.823804 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:29.824098 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:29.824100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:29.824323 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:29.824419 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:29.824426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:29.825199 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:29.825204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:29.825850 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:29.825859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:29.826322 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:29.826325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:29.826416 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:29.827097 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:29.827113 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:29.829200 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":136,"threads_started":1}
W20260810 22:29:29.829502 12189 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:29.823474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:34465 (local address 127.1.221.190:35453)
0810 22:29:29.823633 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:29.823639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:29.823742 (+ 103us) server_negotiation.cc:407] Connection header received
0810 22:29:29.823913 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:29.823916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:29.823973 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:29.824065 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:29.824548 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:29.825082 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:29.825970 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:29.826201 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:29.826313 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:29.829342 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:34465: BlockingRecv error: recv got EOF from 127.1.221.131:34465 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 10 22:29:30 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:30 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:30.429956 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:30.424103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:33675)
0810 22:29:30.424380 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:30.424397 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:30.424509 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:30.424818 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:30.424820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:30.425013 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:30.425110 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:30.425115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:30.426035 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:30.426038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:30.426601 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:30.426609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:30.427167 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:30.427169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:30.427235 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:30.427727 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:30.427739 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:30.429822 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":92,"threads_started":1}
W20260810 22:29:30.430151 12193 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:30.424150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:33675 (local address 127.1.221.190:35453)
0810 22:29:30.424428 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:30.424432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:30.424456 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:29:30.424628 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:30.424631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:30.424688 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:30.424777 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:30.425261 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:30.425930 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:30.426814 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:30.427048 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:30.427137 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:30.429972 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:33675: BlockingRecv error: recv got EOF from 127.1.221.129:33675 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":97,"threads_started":1}
Aug 10 22:29:30 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:30 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:30.626837 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:30.620968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:39075)
0810 22:29:30.621322 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:30.621339 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:30.621479 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:30.621804 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:30.621807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:30.622062 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:30.622184 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:30.622192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:30.622974 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:30.622979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:30.623587 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:30.623595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:30.624095 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:30.624099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:30.624189 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:30.624768 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:30.624788 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:30.626708 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":90,"threads_started":1}
W20260810 22:29:30.627102 12193 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:30.621092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:39075 (local address 127.1.221.190:35453)
0810 22:29:30.621262 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:30.621267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:30.621420 (+ 153us) server_negotiation.cc:407] Connection header received
0810 22:29:30.621621 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:30.621625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:30.621683 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:30.621783 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:30.622298 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:30.622854 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:30.623720 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:30.623971 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:30.624079 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:30.626860 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:39075: BlockingRecv error: recv got EOF from 127.1.221.130:39075 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 22:29:30 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:30 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:30.836055 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:30.830354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:47051)
0810 22:29:30.830728 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:30.830742 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:30.830887 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:30.831189 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:30.831192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:30.831420 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:30.831513 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:30.831518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:30.832290 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:30.832296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:30.832862 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:30.832870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:30.833350 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:30.833354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:30.833442 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:30.834016 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:30.834029 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:30.835934 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
W20260810 22:29:30.836210 12193 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:30.830469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:47051 (local address 127.1.221.190:35453)
0810 22:29:30.830629 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:30.830634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:30.830833 (+ 199us) server_negotiation.cc:407] Connection header received
0810 22:29:30.831003 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:30.831006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:30.831063 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:30.831167 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:30.831636 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:30.832175 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:30.832989 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:30.833226 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:30.833335 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:30.836080 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:47051: BlockingRecv error: recv got EOF from 127.1.221.131:47051 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 22:29:31 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:31 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:31.436555 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:31.430756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:58993)
0810 22:29:31.431054 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:31.431068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:31.431162 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:31.431487 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:31.431489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:31.431692 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:31.431794 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:31.431800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:31.432551 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:31.432554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:31.433120 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:31.433127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:31.433552 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:31.433555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:31.433695 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:31.434259 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:31.434271 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:31.436383 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":108,"threads_started":1}
W20260810 22:29:31.436745 12197 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:31.430986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:58993 (local address 127.1.221.190:35453)
0810 22:29:31.431234 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:31.431238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:31.431252 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:29:31.431305 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:31.431308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:31.431348 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:31.431434 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:31.431914 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:31.432451 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:31.433226 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:31.433452 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:31.433527 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:31.436592 (+ 3065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:58993: BlockingRecv error: recv got EOF from 127.1.221.129:58993 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
Aug 10 22:29:31 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:31 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:31.634093 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:31.627652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:49859)
0810 22:29:31.627929 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:31.627943 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:31.628044 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:31.628289 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:31.628291 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:31.628593 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:31.628731 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:31.628752 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:31.629875 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:31.629877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:31.630430 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:31.630438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:31.630939 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:31.630942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:31.631011 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:31.631523 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:31.631535 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:31.633904 (+ 2369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":97,"threads_started":1}
W20260810 22:29:31.634135 12197 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:31.627737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:49859 (local address 127.1.221.190:35453)
0810 22:29:31.627877 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:31.627881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:31.628013 (+ 132us) server_negotiation.cc:407] Connection header received
0810 22:29:31.628130 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:31.628132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:31.628177 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:31.628251 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:31.628870 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:31.629770 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:31.630529 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:31.630765 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:31.630840 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:31.634034 (+ 3194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:49859: BlockingRecv error: recv got EOF from 127.1.221.130:49859 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 22:29:31 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:31 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:31.842367 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:31.836829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:59343)
0810 22:29:31.837132 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:31.837151 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:31.837241 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:31.837488 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:31.837490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:31.837704 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:31.837802 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:31.837808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:31.838505 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:31.838507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:31.839051 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:31.839058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:31.839441 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:31.839443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:31.839558 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:31.840075 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:31.840086 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:31.842189 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":118,"threads_started":1}
W20260810 22:29:31.842448 12197 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:31.837045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:59343 (local address 127.1.221.190:35453)
0810 22:29:31.837187 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:31.837191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:31.837206 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:29:31.837329 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:31.837331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:31.837376 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:31.837450 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:31.837904 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:31.838410 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:31.839146 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:31.839346 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:31.839417 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:31.842333 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:59343: BlockingRecv error: recv got EOF from 127.1.221.131:59343 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 22:29:32 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:32 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:32.443624 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:32.437672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:44927)
0810 22:29:32.437985 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:32.437999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:32.438096 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:32.438357 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:32.438359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:32.438552 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:32.438650 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:32.438655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:32.439525 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:32.439530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:32.440093 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:32.440101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:32.440599 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:32.440603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:32.440701 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:32.441267 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:32.441285 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:32.443442 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":93,"threads_started":1}
W20260810 22:29:32.443703 12201 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:32.437758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:44927 (local address 127.1.221.190:35453)
0810 22:29:32.438042 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:32.438046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:32.438061 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:29:32.438189 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:32.438191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:32.438235 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:32.438310 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:32.438847 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:32.439400 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:32.440225 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:32.440470 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:32.440583 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:32.443576 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:44927: BlockingRecv error: recv got EOF from 127.1.221.129:44927 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
Aug 10 22:29:32 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:32 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:32.640466 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:32.634962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:55841)
0810 22:29:32.635277 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:32.635291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:32.635390 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:32.635618 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:32.635620 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:32.635819 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:32.635917 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:32.635922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:32.636640 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:32.636642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:32.637189 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:32.637196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:32.637669 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:32.637672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:32.637736 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:32.638296 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:32.638308 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:32.640296 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":125,"threads_started":1}
W20260810 22:29:32.640817 12201 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:32.635117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:55841 (local address 127.1.221.190:35453)
0810 22:29:32.635278 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:32.635282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:32.635343 (+ 61us) server_negotiation.cc:407] Connection header received
0810 22:29:32.635456 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:32.635458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:32.635502 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:32.635571 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:32.636020 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:32.636545 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:32.637294 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:32.637506 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:32.637601 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:32.640456 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:55841: BlockingRecv error: recv got EOF from 127.1.221.130:55841 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 22:29:32 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:32 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:32.848408 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:32.843159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:35151)
0810 22:29:32.843434 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:32.843447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:32.843551 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:32.843772 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:32.843774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:32.843961 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:32.844057 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:32.844063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:32.844773 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:32.844776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:32.845341 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:32.845349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:32.845769 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:32.845772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:32.845841 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:32.846409 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:32.846421 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:32.848280 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
W20260810 22:29:32.848548 12201 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:32.843212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:35151 (local address 127.1.221.190:35453)
0810 22:29:32.843367 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:32.843372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:32.843520 (+ 148us) server_negotiation.cc:407] Connection header received
0810 22:29:32.843618 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:32.843620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:32.843668 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:32.843735 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:32.844156 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:32.844679 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:32.845441 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:32.845672 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:32.845748 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:32.848427 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:35151: BlockingRecv error: recv got EOF from 127.1.221.131:35151 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 22:29:33 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:33 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:33.454880 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:33.444468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:60623)
0810 22:29:33.444802 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:33.444816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:33.444911 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:33.445216 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:33.445218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:33.445415 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:33.445520 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:33.445527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:33.446331 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:33.446333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:33.446895 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:33.446903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:33.447377 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:33.447382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:33.447463 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:33.452500 (+ 5037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:33.452513 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:33.454656 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":128,"threads_started":1}
W20260810 22:29:33.455157 12205 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:33.444643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:60623 (local address 127.1.221.190:35453)
0810 22:29:33.444985 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:33.444988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:33.445002 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:29:33.445041 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:33.445043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:33.445092 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:33.445177 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:33.445669 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:33.446228 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:33.447002 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:33.447231 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:33.447310 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:33.454863 (+ 7553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:60623: BlockingRecv error: recv got EOF from 127.1.221.129:60623 (error 108)
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
Aug 10 22:29:33 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:33 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:33.646909 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:33.641381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:33251)
0810 22:29:33.641690 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:33.641705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:33.641816 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:33.642061 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:33.642063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:33.642262 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:33.642359 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:33.642363 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:33.643125 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:33.643129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:33.643817 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:33.643828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:33.644348 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:33.644351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:33.644414 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:33.644982 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:33.644995 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:33.646794 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":112,"threads_started":1}
W20260810 22:29:33.647188 12205 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:33.641460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:33251 (local address 127.1.221.190:35453)
0810 22:29:33.641662 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:33.641665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:33.641763 (+ 98us) server_negotiation.cc:407] Connection header received
0810 22:29:33.641881 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:33.641884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:33.641926 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:33.642019 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:33.642455 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:33.643014 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:33.643972 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:33.644233 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:33.644331 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:33.646942 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:33251: BlockingRecv error: recv got EOF from 127.1.221.130:33251 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 22:29:33 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:33 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.131@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:33.855090 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:33.849363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.131:51995)
0810 22:29:33.849676 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:33.849689 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:33.849792 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:33.850029 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:33.850031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:33.850219 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:33.850311 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:33.850316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:33.851459 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:33.851465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:33.852088 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:33.852098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:33.852514 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:33.852516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:33.852577 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:33.853048 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:33.853060 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:33.854963 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":117,"threads_started":1}
W20260810 22:29:33.855266 12205 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:33.849419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:51995 (local address 127.1.221.190:35453)
0810 22:29:33.849610 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:33.849616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:33.849759 (+ 143us) server_negotiation.cc:407] Connection header received
0810 22:29:33.849874 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:33.849876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:33.849923 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:33.849994 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:33.850406 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:33.851328 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:33.852197 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:33.852411 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:33.852490 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:33.855107 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.131:51995: BlockingRecv error: recv got EOF from 127.1.221.131:51995 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 22:29:34 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:34 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.129@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:34.461146 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:34.455777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.129:43075)
0810 22:29:34.456096 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:34.456110 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:34.456198 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:34.456433 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:34.456435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:34.456625 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:34.456722 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:34.456727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:34.457444 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:34.457446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:34.458082 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:34.458092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:34.458501 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:34.458503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:34.458564 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:34.459057 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:34.459069 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:34.461024 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":109,"threads_started":1}
W20260810 22:29:34.461351 12211 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:34.455893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:43075 (local address 127.1.221.190:35453)
0810 22:29:34.456148 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:34.456151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:34.456165 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:29:34.456282 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:34.456284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:34.456325 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:34.456393 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:34.456831 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:34.457350 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:34.458187 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:34.458406 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:34.458479 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:34.461174 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.129:43075: BlockingRecv error: recv got EOF from 127.1.221.129:43075 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":64,"threads_started":1}
Aug 10 22:29:34 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:29:34 dist-test-slave-njxd krb5kdc[11036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.221.130@KRBTEST.COM for oryx/127.1.221.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:29:34.653676 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:34.647897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:35453 (local address 127.1.221.130:55173)
0810 22:29:34.648198 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:34.648219 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:34.648359 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:34.648639 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:34.648642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:34.648869 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:34.648976 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:34.648981 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:34.649795 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:34.649799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:34.650381 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:34.650389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:34.650870 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:34.650874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:34.650965 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:34.651551 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:34.651562 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:34.653464 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.221.190:35453: Server oryx/127.1.221.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
W20260810 22:29:34.653733 12211 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:34.647993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:55173 (local address 127.1.221.190:35453)
0810 22:29:34.648163 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:34.648168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:34.648306 (+ 138us) server_negotiation.cc:407] Connection header received
0810 22:29:34.648465 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:34.648468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:34.648524 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:34.648611 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:34.649097 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:34.649682 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:34.650503 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:34.650750 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:34.650865 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:34.653614 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.221.130:55173: BlockingRecv error: recv got EOF from 127.1.221.130:55173 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 22:29:34.743095 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 11689
I20260810 22:29:34.751727 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 11858
I20260810 22:29:34.759845 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 11999
I20260810 22:29:34.767086 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 11608
2026-08-10T22:29:34Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18596 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T22:29:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:34Z Disabled control of system clock
I20260810 22:29:34.801365 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33077
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:33077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 22:29:34.911880 12218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:34.912159 12218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:34.912194 12218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:34.916107 12218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:34.916160 12218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:34.916175 12218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:34.916193 12218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:34.916208 12218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:29:34.921147 12218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:33077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:34.922273 12218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:34.923666 12218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:34.932493 12223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:34.932559 12224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:34.932549 12218 server_base.cc:1034] running on GCE node
W20260810 22:29:34.932498 12227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:34.933326 12218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:34.934478 12218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:34.935678 12218 hybrid_clock.cc:648] HybridClock initialized: now 1786400974935661 us; error 28 us; skew 500 ppm
I20260810 22:29:34.937652 12218 webserver.cc:460] Webserver started at http://127.1.221.190:34915/ using document root <none> and password file <none>
I20260810 22:29:34.938254 12218 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:34.938323 12218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:34.938567 12218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:34.941066 12218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "0c6b37a066214fad95a9df7c98221879"
format_stamp: "Formatted at 2026-08-10 22:29:34 on dist-test-slave-njxd"
server_key: "dcc4f25387a836c3dfd864c3d45555dc"
server_key_iv: "33958dd991aa1d4c2912a4dc528691ef"
server_key_version: "encryptionkey@0"
I20260810 22:29:34.941752 12218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "0c6b37a066214fad95a9df7c98221879"
format_stamp: "Formatted at 2026-08-10 22:29:34 on dist-test-slave-njxd"
server_key: "dcc4f25387a836c3dfd864c3d45555dc"
server_key_iv: "33958dd991aa1d4c2912a4dc528691ef"
server_key_version: "encryptionkey@0"
I20260810 22:29:34.945436 12218 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:29:34.948028 12233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:34.949235 12218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:29:34.949381 12218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "0c6b37a066214fad95a9df7c98221879"
format_stamp: "Formatted at 2026-08-10 22:29:34 on dist-test-slave-njxd"
server_key: "dcc4f25387a836c3dfd864c3d45555dc"
server_key_iv: "33958dd991aa1d4c2912a4dc528691ef"
server_key_version: "encryptionkey@0"
I20260810 22:29:34.949499 12218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:34.964871 12218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:34.965653 12218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:34.965863 12218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:34.973126 12218 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:33077
I20260810 22:29:34.973140 12288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:33077 every 8 connection(s)
I20260810 22:29:34.974167 12218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:29:34.977007 12289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:34.977449 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 12218
I20260810 22:29:34.977581 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:29:34.977890 1910 external_mini_cluster.cc:1428] Setting key f6eed879ad821ce9f5f24ee9fe7f7ff6
I20260810 22:29:34.983053 12289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879: Bootstrap starting.
I20260810 22:29:34.985250 12289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:34.986083 12289 log.cc:824] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:34.987100 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:34.979280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.1.221.190:33077)
0810 22:29:34.979914 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:34.979929 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:34.979963 (+ 34us) server_negotiation.cc:407] Connection header received
0810 22:29:34.980617 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:34.980634 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:34.981016 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:34.981365 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:34.981816 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:34.982748 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:34.983703 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:34.984169 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:34.984351 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:34.985223 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:34.985228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:34.985242 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:34.985762 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:34.985825 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:34.985847 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:34.985962 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:34.986083 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 22:29:34.986374 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":140,"threads_started":1}
I20260810 22:29:34.988273 12289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879: No bootstrap required, opened a new log
I20260810 22:29:34.991834 12289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6b37a066214fad95a9df7c98221879" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33077 } }
I20260810 22:29:34.992072 12289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:34.992161 12289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6b37a066214fad95a9df7c98221879, State: Initialized, Role: FOLLOWER
I20260810 22:29:34.992578 12289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6b37a066214fad95a9df7c98221879" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33077 } }
I20260810 22:29:34.992725 12289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:29:34.992812 12289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:29:34.992930 12289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:34.993964 12289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6b37a066214fad95a9df7c98221879" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33077 } }
I20260810 22:29:34.994352 12289 leader_election.cc:302] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c6b37a066214fad95a9df7c98221879; no voters:
I20260810 22:29:34.994709 12289 leader_election.cc:288] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:29:34.994863 12296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:34.995115 12296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [term 1 LEADER]: Becoming Leader. State: Replica: 0c6b37a066214fad95a9df7c98221879, State: Running, Role: LEADER
I20260810 22:29:34.995446 12296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6b37a066214fad95a9df7c98221879" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33077 } }
I20260810 22:29:34.995959 12289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:34.997170 12297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c6b37a066214fad95a9df7c98221879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6b37a066214fad95a9df7c98221879" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33077 } } }
I20260810 22:29:34.997227 12298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c6b37a066214fad95a9df7c98221879. Latest consensus state: current_term: 1 leader_uuid: "0c6b37a066214fad95a9df7c98221879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6b37a066214fad95a9df7c98221879" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33077 } } }
I20260810 22:29:34.997356 12298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:34.997303 12297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:34.997844 12306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:35.001133 12306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:35.006532 12306 catalog_manager.cc:1347] Generated new cluster ID: 7104c1b73c4d4e08b9ebf17c0d134770
I20260810 22:29:35.006611 12306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:35.020803 12306 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:29:35.021816 12306 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:35.030632 12306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c6b37a066214fad95a9df7c98221879: Generated new TSK 0
I20260810 22:29:35.031319 12306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:29:35.040098 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33077
--builtin_ntp_servers=127.1.221.148:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 22:29:35.161103 12318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:35.161351 12318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:35.161388 12318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:35.165544 12318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:35.165683 12318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:35.165741 12318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:29:35.170567 12318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:35.171710 12318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:35.173166 12318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:35.183223 12327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:35.183218 12325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:35.183383 12318 server_base.cc:1034] running on GCE node
W20260810 22:29:35.183223 12324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:35.183949 12318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:35.184585 12318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:35.185772 12318 hybrid_clock.cc:648] HybridClock initialized: now 1786400975185750 us; error 32 us; skew 500 ppm
I20260810 22:29:35.187597 12318 webserver.cc:460] Webserver started at http://127.1.221.129:40391/ using document root <none> and password file <none>
I20260810 22:29:35.188166 12318 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:35.188259 12318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:35.188530 12318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:35.190680 12318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "9e3147cc28624ac6b7d56e83940b3010"
format_stamp: "Formatted at 2026-08-10 22:29:35 on dist-test-slave-njxd"
server_key: "b5b0fe17379f61d0e0f139046a9529d8"
server_key_iv: "555ef8208c46eb6a9663803be8b5d95f"
server_key_version: "encryptionkey@0"
I20260810 22:29:35.191311 12318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "9e3147cc28624ac6b7d56e83940b3010"
format_stamp: "Formatted at 2026-08-10 22:29:35 on dist-test-slave-njxd"
server_key: "b5b0fe17379f61d0e0f139046a9529d8"
server_key_iv: "555ef8208c46eb6a9663803be8b5d95f"
server_key_version: "encryptionkey@0"
I20260810 22:29:35.195048 12318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:29:35.197871 12333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:35.199069 12318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260810 22:29:35.199211 12318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "9e3147cc28624ac6b7d56e83940b3010"
format_stamp: "Formatted at 2026-08-10 22:29:35 on dist-test-slave-njxd"
server_key: "b5b0fe17379f61d0e0f139046a9529d8"
server_key_iv: "555ef8208c46eb6a9663803be8b5d95f"
server_key_version: "encryptionkey@0"
I20260810 22:29:35.199337 12318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:35.210903 12318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:35.211637 12318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:35.211843 12318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:35.212447 12318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:35.213403 12318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:35.213476 12318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:35.213551 12318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:35.213624 12318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:35.222995 12318 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:33651
I20260810 22:29:35.223021 12466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:33651 every 8 connection(s)
I20260810 22:29:35.223955 12318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:29:35.227890 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 12318
I20260810 22:29:35.228016 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:29:35.228389 1910 external_mini_cluster.cc:1428] Setting key 9f9ad43d1db54bfacadb132e40bf03f2
I20260810 22:29:35.230600 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33077
--builtin_ntp_servers=127.1.221.148:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 22:29:35.234835 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:35.225814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:56709 (local address 127.1.221.190:33077)
0810 22:29:35.225995 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:35.226000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:35.227175 (+ 1175us) server_negotiation.cc:407] Connection header received
0810 22:29:35.228992 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:35.228995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:35.229053 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:35.229136 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:35.230183 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:35.230819 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:35.232615 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:35.232985 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:35.233101 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:35.234230 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:35.234236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:35.234243 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:35.234260 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:35.234294 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:35.234307 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:35.234387 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:35.234540 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 22:29:35.234629 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:29:35.235620 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:35.226336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33077 (local address 127.1.221.129:56709)
0810 22:29:35.227018 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:35.227060 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:35.228746 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:35.229318 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:35.229325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:35.229410 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:35.230027 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:35.230045 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:35.230955 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:35.230962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:35.232469 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:35.232491 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:35.233147 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:35.233150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:35.233285 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:35.233947 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:35.233949 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:35.233988 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:35.234053 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:35.234395 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:35.234402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:35.234406 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 22:29:35.234494 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 22:29:35.234723 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260810 22:29:35.236881 12467 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33077
I20260810 22:29:35.237094 12467 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:35.237637 12467 heartbeater.cc:511] Master 127.1.221.190:33077 requested a full tablet report, sending...
I20260810 22:29:35.239077 12251 ts_manager.cc:194] Registered new tserver with Master: 9e3147cc28624ac6b7d56e83940b3010 (127.1.221.129:33651)
I20260810 22:29:35.240164 12251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.129:56709
W20260810 22:29:35.344471 12470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:35.344722 12470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:35.344759 12470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:35.348937 12470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:35.348994 12470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:35.349080 12470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:29:35.353899 12470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:35.355041 12470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:35.356463 12470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:29:35.366349 12470 server_base.cc:1034] running on GCE node
W20260810 22:29:35.366334 12479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:35.366346 12477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:35.366370 12476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:35.367005 12470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:35.367614 12470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:35.368818 12470 hybrid_clock.cc:648] HybridClock initialized: now 1786400975368804 us; error 25 us; skew 500 ppm
I20260810 22:29:35.370723 12470 webserver.cc:460] Webserver started at http://127.1.221.130:45117/ using document root <none> and password file <none>
I20260810 22:29:35.371333 12470 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:35.371431 12470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:35.371670 12470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:35.373931 12470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "41161482ae2e4cca9fa695a80ca85651"
format_stamp: "Formatted at 2026-08-10 22:29:35 on dist-test-slave-njxd"
server_key: "e58eb513a7bb5520b7fa77d2fc4e79ea"
server_key_iv: "4fdeabfed9c8efb4f2aec117d9cd9e40"
server_key_version: "encryptionkey@0"
I20260810 22:29:35.374540 12470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "41161482ae2e4cca9fa695a80ca85651"
format_stamp: "Formatted at 2026-08-10 22:29:35 on dist-test-slave-njxd"
server_key: "e58eb513a7bb5520b7fa77d2fc4e79ea"
server_key_iv: "4fdeabfed9c8efb4f2aec117d9cd9e40"
server_key_version: "encryptionkey@0"
I20260810 22:29:35.378532 12470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:29:35.381158 12485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:35.382449 12470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260810 22:29:35.382596 12470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "41161482ae2e4cca9fa695a80ca85651"
format_stamp: "Formatted at 2026-08-10 22:29:35 on dist-test-slave-njxd"
server_key: "e58eb513a7bb5520b7fa77d2fc4e79ea"
server_key_iv: "4fdeabfed9c8efb4f2aec117d9cd9e40"
server_key_version: "encryptionkey@0"
I20260810 22:29:35.382714 12470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:35.393546 12470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:35.394307 12470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:35.394505 12470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:35.395108 12470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:35.396130 12470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:35.396199 12470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:35.396275 12470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:35.396324 12470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:35.405664 12470 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:38385
I20260810 22:29:35.405690 12612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:38385 every 8 connection(s)
I20260810 22:29:35.406643 12470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:29:35.408119 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 12470
I20260810 22:29:35.408215 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:29:35.408607 1910 external_mini_cluster.cc:1428] Setting key cfa49f398d917f0a9dd05df8d66453c0
I20260810 22:29:35.410735 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33077
--builtin_ntp_servers=127.1.221.148:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 22:29:35.416627 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:35.408692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:45265 (local address 127.1.221.190:33077)
0810 22:29:35.409716 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:35.409721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:35.410112 (+ 391us) server_negotiation.cc:407] Connection header received
0810 22:29:35.411646 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:35.411649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:35.411707 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:35.411804 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:35.412753 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:35.413294 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:35.414579 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:35.414815 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:35.414934 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:35.416134 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:35.416138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:35.416140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:35.416153 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:35.416172 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:35.416180 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:35.416260 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:35.416421 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 22:29:35.416484 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":936}
I20260810 22:29:35.417522 12616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:35.409138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33077 (local address 127.1.221.130:45265)
0810 22:29:35.409965 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:35.410019 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:35.411412 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:35.412008 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:35.412015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:35.412101 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:35.412603 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:35.412615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:35.413443 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:35.413447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:35.414430 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:35.414441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:35.414982 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:35.414986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:35.415162 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:35.415857 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:35.415858 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:35.415901 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:35.415965 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:35.416262 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:35.416273 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:35.416281 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:29:35.416421 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 22:29:35.416714 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":142,"threads_started":1}
I20260810 22:29:35.418787 12613 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33077
I20260810 22:29:35.419072 12613 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:35.419798 12613 heartbeater.cc:511] Master 127.1.221.190:33077 requested a full tablet report, sending...
I20260810 22:29:35.421096 12251 ts_manager.cc:194] Registered new tserver with Master: 41161482ae2e4cca9fa695a80ca85651 (127.1.221.130:38385)
I20260810 22:29:35.422021 12251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.130:45265
W20260810 22:29:35.530673 12617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:35.530987 12617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:35.531028 12617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:35.535274 12617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:35.535339 12617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:35.535398 12617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:29:35.540315 12617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:35.541416 12617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:35.542900 12617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:35.552922 12627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:35.552932 12624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:35.552932 12625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:35.553356 12617 server_base.cc:1034] running on GCE node
I20260810 22:29:35.553896 12617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:35.554621 12617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:35.555817 12617 hybrid_clock.cc:648] HybridClock initialized: now 1786400975555805 us; error 33 us; skew 500 ppm
I20260810 22:29:35.557915 12617 webserver.cc:460] Webserver started at http://127.1.221.131:45247/ using document root <none> and password file <none>
I20260810 22:29:35.558766 12617 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:35.558897 12617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:35.559163 12617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:35.561920 12617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "7f1e677bd5ea4caaaef8915c43ccc188"
format_stamp: "Formatted at 2026-08-10 22:29:35 on dist-test-slave-njxd"
server_key: "25fa1f9e818071941862a8a07e74ba23"
server_key_iv: "a434f294cf98e11fb8fde97caa33367c"
server_key_version: "encryptionkey@0"
I20260810 22:29:35.562551 12617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "7f1e677bd5ea4caaaef8915c43ccc188"
format_stamp: "Formatted at 2026-08-10 22:29:35 on dist-test-slave-njxd"
server_key: "25fa1f9e818071941862a8a07e74ba23"
server_key_iv: "a434f294cf98e11fb8fde97caa33367c"
server_key_version: "encryptionkey@0"
I20260810 22:29:35.566578 12617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:29:35.569393 12633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:35.570817 12617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:29:35.570972 12617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "7f1e677bd5ea4caaaef8915c43ccc188"
format_stamp: "Formatted at 2026-08-10 22:29:35 on dist-test-slave-njxd"
server_key: "25fa1f9e818071941862a8a07e74ba23"
server_key_iv: "a434f294cf98e11fb8fde97caa33367c"
server_key_version: "encryptionkey@0"
I20260810 22:29:35.571110 12617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:35.593062 12617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:35.593977 12617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:35.594189 12617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:35.594866 12617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:35.595872 12617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:35.595947 12617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:35.596019 12617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:35.596071 12617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:35.605975 12617 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:35321
I20260810 22:29:35.606001 12765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:35321 every 8 connection(s)
I20260810 22:29:35.606935 12617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:29:35.608407 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 12617
I20260810 22:29:35.608520 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:29:35.608844 1910 external_mini_cluster.cc:1428] Setting key 0fd035b4abaa5bbe3248828a545e9009
I20260810 22:29:35.616219 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:35.608911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:42817 (local address 127.1.221.190:33077)
0810 22:29:35.609061 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:35.609065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:35.610363 (+ 1298us) server_negotiation.cc:407] Connection header received
0810 22:29:35.611733 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:35.611736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:35.611789 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:35.611857 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:35.612743 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:35.613262 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:35.614454 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:35.614681 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:35.614759 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:35.615805 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:35.615808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:35.615810 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:35.615822 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:35.615842 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:35.615850 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:35.615908 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:35.616053 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 22:29:35.616104 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:29:35.617072 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:35.609389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33077 (local address 127.1.221.131:42817)
0810 22:29:35.610193 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:35.610247 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:35.611537 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:35.612032 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:35.612039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:35.612125 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:35.612621 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:35.612633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:35.613369 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:35.613371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:35.614347 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:35.614356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:35.614815 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:35.614818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:35.614961 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:35.615538 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:35.615539 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:35.615580 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:35.615645 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:35.615924 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:35.615931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:35.615935 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 22:29:35.616031 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 22:29:35.616252 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":198,"threads_started":1}
I20260810 22:29:35.618238 12766 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33077
I20260810 22:29:35.618495 12766 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:35.619038 12766 heartbeater.cc:511] Master 127.1.221.190:33077 requested a full tablet report, sending...
I20260810 22:29:35.620054 12251 ts_manager.cc:194] Registered new tserver with Master: 7f1e677bd5ea4caaaef8915c43ccc188 (127.1.221.131:35321)
I20260810 22:29:35.620625 12251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.131:42817
I20260810 22:29:35.622838 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 22:29:35.625674 12294 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:35.624764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.1.221.190:33077)
0810 22:29:35.624881 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:35.624885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:35.625076 (+ 191us) server_negotiation.cc:407] Connection header received
0810 22:29:35.625172 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:35.625175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:35.625213 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:35.625281 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:35.625515 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33774: BlockingRecv error: recv got EOF from 127.0.0.1:33774 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:29:35.625864 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 12318
I20260810 22:29:35.638703 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 12470
I20260810 22:29:35.647930 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 12617
I20260810 22:29:35.654692 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 12218
2026-08-10T22:29:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (886 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T22:29:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:35Z Disabled control of system clock
I20260810 22:29:35.687747 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39177
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:39177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 22:29:35.804177 12778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:35.804448 12778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:35.804486 12778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:35.808507 12778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:35.808575 12778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:35.808591 12778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:35.808609 12778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:35.808624 12778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:29:35.813349 12778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:39177
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:35.814492 12778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:35.815910 12778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:35.824566 12787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:35.824558 12783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:35.824570 12785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:35.824811 12778 server_base.cc:1034] running on GCE node
I20260810 22:29:35.825472 12778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:35.826637 12778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:35.827814 12778 hybrid_clock.cc:648] HybridClock initialized: now 1786400975827779 us; error 44 us; skew 500 ppm
I20260810 22:29:35.830209 12778 webserver.cc:460] Webserver started at http://127.1.221.190:36869/ using document root <none> and password file <none>
I20260810 22:29:35.830803 12778 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:35.830868 12778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:35.831135 12778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:35.833215 12778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "d3717cd5175c40b79272ef219f51f5e2"
format_stamp: "Formatted at 2026-08-10 22:29:35 on dist-test-slave-njxd"
server_key: "fbf910d2a0d33aaabc827a8ab6cfacd4"
server_key_iv: "b1ec8504281c649b6690d394a7735b82"
server_key_version: "encryptionkey@0"
I20260810 22:29:35.833889 12778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "d3717cd5175c40b79272ef219f51f5e2"
format_stamp: "Formatted at 2026-08-10 22:29:35 on dist-test-slave-njxd"
server_key: "fbf910d2a0d33aaabc827a8ab6cfacd4"
server_key_iv: "b1ec8504281c649b6690d394a7735b82"
server_key_version: "encryptionkey@0"
I20260810 22:29:35.837512 12778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:29:35.840075 12794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:35.841364 12778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:29:35.841495 12778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "d3717cd5175c40b79272ef219f51f5e2"
format_stamp: "Formatted at 2026-08-10 22:29:35 on dist-test-slave-njxd"
server_key: "fbf910d2a0d33aaabc827a8ab6cfacd4"
server_key_iv: "b1ec8504281c649b6690d394a7735b82"
server_key_version: "encryptionkey@0"
I20260810 22:29:35.841671 12778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:35.846573 12778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:35.847229 12778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:35.847422 12778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:35.855057 12778 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:39177
I20260810 22:29:35.855062 12856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:39177 every 8 connection(s)
I20260810 22:29:35.856065 12778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:29:35.859404 12858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:35.864257 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 12778
I20260810 22:29:35.864367 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:29:35.864650 1910 external_mini_cluster.cc:1428] Setting key d1d33af88af9108096a850a09ce586fe
I20260810 22:29:35.865368 12858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2: Bootstrap starting.
I20260810 22:29:35.867877 12858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:35.868710 12858 log.cc:824] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:35.870456 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:35.866165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.1.221.190:39177)
0810 22:29:35.866855 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:35.866870 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:35.866906 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:29:35.867538 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:35.867555 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:35.867919 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:35.868308 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:35.869153 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:35.869158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:35.869169 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:35.869179 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:35.869243 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:35.869259 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:35.869376 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:35.869483 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 22:29:35.869780 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":140,"threads_started":1}
I20260810 22:29:35.870816 12858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2: No bootstrap required, opened a new log
I20260810 22:29:35.874491 12858 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3717cd5175c40b79272ef219f51f5e2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39177 } }
I20260810 22:29:35.874755 12858 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:35.874842 12858 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3717cd5175c40b79272ef219f51f5e2, State: Initialized, Role: FOLLOWER
I20260810 22:29:35.875335 12858 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3717cd5175c40b79272ef219f51f5e2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39177 } }
I20260810 22:29:35.875509 12858 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:29:35.875573 12858 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:29:35.875676 12858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:35.876631 12858 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3717cd5175c40b79272ef219f51f5e2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39177 } }
I20260810 22:29:35.876981 12858 leader_election.cc:302] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3717cd5175c40b79272ef219f51f5e2; no voters:
I20260810 22:29:35.877327 12858 leader_election.cc:288] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:29:35.877445 12863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:35.877728 12863 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [term 1 LEADER]: Becoming Leader. State: Replica: d3717cd5175c40b79272ef219f51f5e2, State: Running, Role: LEADER
I20260810 22:29:35.878067 12863 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3717cd5175c40b79272ef219f51f5e2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39177 } }
I20260810 22:29:35.878623 12858 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:35.879832 12864 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3717cd5175c40b79272ef219f51f5e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3717cd5175c40b79272ef219f51f5e2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39177 } } }
I20260810 22:29:35.879832 12865 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3717cd5175c40b79272ef219f51f5e2. Latest consensus state: current_term: 1 leader_uuid: "d3717cd5175c40b79272ef219f51f5e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3717cd5175c40b79272ef219f51f5e2" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39177 } } }
I20260810 22:29:35.879989 12865 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:35.879989 12864 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2 [sys.catalog]: This master's current role is: LEADER
W20260810 22:29:35.883618 12880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:29:35.883718 12880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:29:35.883812 12872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:35.884538 12872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:35.889727 12872 catalog_manager.cc:1347] Generated new cluster ID: bb28cadfc92a460aa3b66646ec6b217a
I20260810 22:29:35.889802 12872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:35.904978 12872 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:29:35.905339 12872 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:35.909873 12872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3717cd5175c40b79272ef219f51f5e2: Generated new TSK 0
I20260810 22:29:35.910681 12872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:29:35.914631 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39177
--builtin_ntp_servers=127.1.221.148:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 22:29:36.032277 12884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:36.032575 12884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:36.032665 12884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:36.036777 12884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:36.036871 12884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:36.036960 12884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:29:36.041879 12884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:36.043025 12884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:36.044498 12884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:36.054494 12893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:36.054510 12891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:36.054491 12890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:36.054749 12884 server_base.cc:1034] running on GCE node
I20260810 22:29:36.055241 12884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:36.055846 12884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:36.057004 12884 hybrid_clock.cc:648] HybridClock initialized: now 1786400976056973 us; error 46 us; skew 500 ppm
I20260810 22:29:36.058907 12884 webserver.cc:460] Webserver started at http://127.1.221.129:42701/ using document root <none> and password file <none>
I20260810 22:29:36.059506 12884 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:36.059568 12884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:36.059809 12884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:36.062112 12884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "d6456fabc3114b48a2b2210676190acf"
format_stamp: "Formatted at 2026-08-10 22:29:36 on dist-test-slave-njxd"
server_key: "0d9a059192a47f98ce1d4aa39c09435e"
server_key_iv: "f614b520de85dc08639ea233995287f9"
server_key_version: "encryptionkey@0"
I20260810 22:29:36.062696 12884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "d6456fabc3114b48a2b2210676190acf"
format_stamp: "Formatted at 2026-08-10 22:29:36 on dist-test-slave-njxd"
server_key: "0d9a059192a47f98ce1d4aa39c09435e"
server_key_iv: "f614b520de85dc08639ea233995287f9"
server_key_version: "encryptionkey@0"
I20260810 22:29:36.066484 12884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:29:36.069178 12900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:36.070574 12884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:29:36.070688 12884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "d6456fabc3114b48a2b2210676190acf"
format_stamp: "Formatted at 2026-08-10 22:29:36 on dist-test-slave-njxd"
server_key: "0d9a059192a47f98ce1d4aa39c09435e"
server_key_iv: "f614b520de85dc08639ea233995287f9"
server_key_version: "encryptionkey@0"
I20260810 22:29:36.070811 12884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:36.075656 12884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:36.076261 12884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:36.076413 12884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:36.076930 12884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:36.077955 12884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:36.078003 12884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:36.078069 12884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:36.078099 12884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:36.087277 12884 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:46785
I20260810 22:29:36.087306 13028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:46785 every 8 connection(s)
I20260810 22:29:36.088224 12884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:29:36.092343 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 12884
I20260810 22:29:36.092438 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:29:36.092798 1910 external_mini_cluster.cc:1428] Setting key 27b02fbbb88e55b2e4376089b6236974
I20260810 22:29:36.094601 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:36.090125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:43983 (local address 127.1.221.190:39177)
0810 22:29:36.090286 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:36.090290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:36.091360 (+ 1070us) server_negotiation.cc:407] Connection header received
0810 22:29:36.092718 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:36.092724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:36.092770 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:36.092872 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:36.094081 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:36.094084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:36.094087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:36.094100 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:36.094129 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:36.094139 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:36.094232 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:36.094407 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 22:29:36.094454 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 22:29:36.094877 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39177
--builtin_ntp_servers=127.1.221.148:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 22:29:36.095587 13031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:36.090535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39177 (local address 127.1.221.129:43983)
0810 22:29:36.091202 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:36.091243 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:36.092502 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:36.093079 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:36.093086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:36.093174 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:36.093814 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:36.093815 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:36.093856 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:36.093925 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:36.094220 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:36.094230 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:36.094236 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:29:36.094452 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 22:29:36.094715 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":125,"threads_started":1}
I20260810 22:29:36.096894 13029 heartbeater.cc:347] Connected to a master server at 127.1.221.190:39177
I20260810 22:29:36.097163 13029 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:36.097673 13029 heartbeater.cc:511] Master 127.1.221.190:39177 requested a full tablet report, sending...
I20260810 22:29:36.099409 12815 ts_manager.cc:194] Registered new tserver with Master: d6456fabc3114b48a2b2210676190acf (127.1.221.129:46785)
W20260810 22:29:36.210273 13033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:36.210522 13033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:36.210561 13033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:36.214481 13033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:36.214526 13033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:36.214594 13033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:29:36.219005 13033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:36.219946 13033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:36.221200 13033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:36.230329 13041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:36.230330 13039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:36.230603 13033 server_base.cc:1034] running on GCE node
W20260810 22:29:36.230376 13038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:36.231022 13033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:36.231647 13033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:36.232808 13033 hybrid_clock.cc:648] HybridClock initialized: now 1786400976232790 us; error 30 us; skew 500 ppm
I20260810 22:29:36.234755 13033 webserver.cc:460] Webserver started at http://127.1.221.130:33391/ using document root <none> and password file <none>
I20260810 22:29:36.235342 13033 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:36.235406 13033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:36.235695 13033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:36.238159 13033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "065f69dfde2c49caa897516371d397eb"
format_stamp: "Formatted at 2026-08-10 22:29:36 on dist-test-slave-njxd"
server_key: "37f610f6f09525074fbbfe20c81235cc"
server_key_iv: "7c6f3b3f6eced42ad00671d1a85b6ada"
server_key_version: "encryptionkey@0"
I20260810 22:29:36.238744 13033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "065f69dfde2c49caa897516371d397eb"
format_stamp: "Formatted at 2026-08-10 22:29:36 on dist-test-slave-njxd"
server_key: "37f610f6f09525074fbbfe20c81235cc"
server_key_iv: "7c6f3b3f6eced42ad00671d1a85b6ada"
server_key_version: "encryptionkey@0"
I20260810 22:29:36.242501 13033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:29:36.244979 13047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:36.246325 13033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:29:36.246445 13033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "065f69dfde2c49caa897516371d397eb"
format_stamp: "Formatted at 2026-08-10 22:29:36 on dist-test-slave-njxd"
server_key: "37f610f6f09525074fbbfe20c81235cc"
server_key_iv: "7c6f3b3f6eced42ad00671d1a85b6ada"
server_key_version: "encryptionkey@0"
I20260810 22:29:36.246577 13033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:36.251225 13033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:36.251858 13033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:36.252046 13033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:36.252636 13033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:36.253671 13033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:36.253720 13033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:36.253784 13033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:36.253822 13033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:36.262758 13033 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:32911
I20260810 22:29:36.262781 13173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:32911 every 8 connection(s)
I20260810 22:29:36.263792 13033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:29:36.270092 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:36.265690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:33233 (local address 127.1.221.190:39177)
0810 22:29:36.265837 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:36.265841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:36.266994 (+ 1153us) server_negotiation.cc:407] Connection header received
0810 22:29:36.268335 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:36.268339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:36.268383 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:36.268496 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:36.269596 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:36.269599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:36.269602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:36.269614 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:36.269637 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:36.269652 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:36.269752 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:36.269901 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 22:29:36.269948 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:29:36.270977 13177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:36.266146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39177 (local address 127.1.221.130:33233)
0810 22:29:36.266837 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:36.266880 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:36.268120 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:36.268675 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:36.268682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:36.268770 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:36.269313 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:36.269315 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:36.269355 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:36.269424 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:36.269746 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:36.269757 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:36.269765 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:29:36.269946 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 22:29:36.270191 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":125,"threads_started":1}
I20260810 22:29:36.272127 13175 heartbeater.cc:347] Connected to a master server at 127.1.221.190:39177
I20260810 22:29:36.272360 13175 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:36.272775 13175 heartbeater.cc:511] Master 127.1.221.190:39177 requested a full tablet report, sending...
I20260810 22:29:36.273350 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 13033
I20260810 22:29:36.273499 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:29:36.273908 1910 external_mini_cluster.cc:1428] Setting key 1ddc3adcdabf0f2d6591d40ae2381fe6
I20260810 22:29:36.273917 12815 ts_manager.cc:194] Registered new tserver with Master: 065f69dfde2c49caa897516371d397eb (127.1.221.130:32911)
I20260810 22:29:36.275780 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39177
--builtin_ntp_servers=127.1.221.148:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 22:29:36.389348 13178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:36.389725 13178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:36.389827 13178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:36.393862 13178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:36.393934 13178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:36.394019 13178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:29:36.398653 13178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:39177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:36.399775 13178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:36.401125 13178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:36.410656 13184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:36.410643 13186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:36.410622 13183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:36.411046 13178 server_base.cc:1034] running on GCE node
I20260810 22:29:36.411443 13178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:36.411981 13178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:36.413165 13178 hybrid_clock.cc:648] HybridClock initialized: now 1786400976413146 us; error 39 us; skew 500 ppm
I20260810 22:29:36.415035 13178 webserver.cc:460] Webserver started at http://127.1.221.131:45319/ using document root <none> and password file <none>
I20260810 22:29:36.415609 13178 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:36.415704 13178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:36.415943 13178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:36.418056 13178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "8817fe7b04e846e7a781f4c48d618222"
format_stamp: "Formatted at 2026-08-10 22:29:36 on dist-test-slave-njxd"
server_key: "276e595d254c7b86ea8473593b897f61"
server_key_iv: "1fded1a69ab9f0eacbd1ddac98a0d355"
server_key_version: "encryptionkey@0"
I20260810 22:29:36.418625 13178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "8817fe7b04e846e7a781f4c48d618222"
format_stamp: "Formatted at 2026-08-10 22:29:36 on dist-test-slave-njxd"
server_key: "276e595d254c7b86ea8473593b897f61"
server_key_iv: "1fded1a69ab9f0eacbd1ddac98a0d355"
server_key_version: "encryptionkey@0"
I20260810 22:29:36.422267 13178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:29:36.424782 13192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:36.426156 13178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:29:36.426303 13178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "8817fe7b04e846e7a781f4c48d618222"
format_stamp: "Formatted at 2026-08-10 22:29:36 on dist-test-slave-njxd"
server_key: "276e595d254c7b86ea8473593b897f61"
server_key_iv: "1fded1a69ab9f0eacbd1ddac98a0d355"
server_key_version: "encryptionkey@0"
I20260810 22:29:36.426429 13178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:36.431097 13178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:36.431725 13178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:36.431941 13178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:36.432688 13178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:36.433760 13178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:36.433851 13178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:36.433915 13178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:36.433966 13178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:36.443038 13178 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:32851
I20260810 22:29:36.443104 13326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:32851 every 8 connection(s)
I20260810 22:29:36.444060 13178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:29:36.450690 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:36.446126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:33247 (local address 127.1.221.190:39177)
0810 22:29:36.446276 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:36.446280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:36.447459 (+ 1179us) server_negotiation.cc:407] Connection header received
0810 22:29:36.448801 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:36.448804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:36.448844 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:36.448922 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:36.450199 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:36.450202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:36.450204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:29:36.450225 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:36.450248 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:29:36.450257 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:36.450337 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:36.450493 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 22:29:36.450548 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:29:36.451633 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:36.446619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39177 (local address 127.1.221.131:33247)
0810 22:29:36.447318 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:36.447362 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:36.448582 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:36.449089 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:36.449096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:36.449223 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:36.449910 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:29:36.449911 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:36.449973 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:29:36.450046 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:36.450317 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:36.450321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:36.450325 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 22:29:36.450515 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 22:29:36.450784 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":127,"threads_started":1}
I20260810 22:29:36.452795 13327 heartbeater.cc:347] Connected to a master server at 127.1.221.190:39177
I20260810 22:29:36.453043 13327 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:36.453431 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 13178
I20260810 22:29:36.453547 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:29:36.453567 13327 heartbeater.cc:511] Master 127.1.221.190:39177 requested a full tablet report, sending...
I20260810 22:29:36.453933 1910 external_mini_cluster.cc:1428] Setting key 0d4473770f6651acc0ae597311a3554b
I20260810 22:29:36.454952 12813 ts_manager.cc:194] Registered new tserver with Master: 8817fe7b04e846e7a781f4c48d618222 (127.1.221.131:32851)
I20260810 22:29:36.456097 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 22:29:36.459256 12861 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:29:36.458335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42206 (local address 127.1.221.190:39177)
0810 22:29:36.458485 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:36.458490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:36.458638 (+ 148us) server_negotiation.cc:407] Connection header received
0810 22:29:36.458747 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:36.458750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:36.458783 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:36.458848 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:36.459128 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42206: BlockingRecv error: recv got EOF from 127.0.0.1:42206 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:29:36.459403 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 12884
I20260810 22:29:36.472118 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 13033
I20260810 22:29:36.482050 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 13178
I20260810 22:29:36.491182 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 12778
2026-08-10T22:29:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (837 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:29:36 dist-test-slave-njxd krb5kdc[13337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:29:36 dist-test-slave-njxd krb5kdc[13337](info): set up 2 sockets
Aug 10 22:29:36 dist-test-slave-njxd krb5kdc[13337](info): commencing operation
krb5kdc: starting...
W20260810 22:29:37.602160 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:29:37 dist-test-slave-njxd krb5kdc[13337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:29:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:29:37.733319 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42475
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:42475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:37.844980 13354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:37.845340 13354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:37.845433 13354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:37.849617 13354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:29:37.849720 13354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:37.849759 13354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:37.849820 13354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:29:37.849858 13354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:29:37.855486 13354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:42475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:37.856654 13354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:37.857786 13354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:37.866366 13360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:37.866380 13359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:37.866381 13362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:37.866591 13354 server_base.cc:1034] running on GCE node
I20260810 22:29:37.867285 13354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:37.868389 13354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:37.869593 13354 hybrid_clock.cc:648] HybridClock initialized: now 1786400977869553 us; error 30 us; skew 500 ppm
Aug 10 22:29:37 dist-test-slave-njxd krb5kdc[13337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400977, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:37.872900 13354 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:29:37.873898 13354 webserver.cc:460] Webserver started at http://127.1.221.190:40063/ using document root <none> and password file <none>
I20260810 22:29:37.874481 13354 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:37.874586 13354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:37.874831 13354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:37.876911 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "133e6ea470714266ae55aed838995d04"
format_stamp: "Formatted at 2026-08-10 22:29:37 on dist-test-slave-njxd"
server_key: "e89df53098ce34564f315c422413e819"
server_key_iv: "f3ef8c7ddff014ad3e147fa976be4913"
server_key_version: "encryptionkey@0"
I20260810 22:29:37.877499 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "133e6ea470714266ae55aed838995d04"
format_stamp: "Formatted at 2026-08-10 22:29:37 on dist-test-slave-njxd"
server_key: "e89df53098ce34564f315c422413e819"
server_key_iv: "f3ef8c7ddff014ad3e147fa976be4913"
server_key_version: "encryptionkey@0"
I20260810 22:29:37.881281 13354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:29:37.883806 13372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:37.884999 13354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:29:37.885144 13354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "133e6ea470714266ae55aed838995d04"
format_stamp: "Formatted at 2026-08-10 22:29:37 on dist-test-slave-njxd"
server_key: "e89df53098ce34564f315c422413e819"
server_key_iv: "f3ef8c7ddff014ad3e147fa976be4913"
server_key_version: "encryptionkey@0"
I20260810 22:29:37.885253 13354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:37.906499 13354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:37.909237 13354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:37.909446 13354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:37.916231 13354 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:42475
I20260810 22:29:37.916536 13428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:42475 every 8 connection(s)
I20260810 22:29:37.917198 13354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:29:37.919945 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:37.920470 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 13354
I20260810 22:29:37.920614 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:29:37.920898 1910 external_mini_cluster.cc:1428] Setting key c2b7df1ab2e41e7c651b76680e39c233
I20260810 22:29:37.926725 13429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04: Bootstrap starting.
I20260810 22:29:37.929464 13429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:37.930348 13429 log.cc:824] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:29:37 dist-test-slave-njxd krb5kdc[13337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:29:37.932542 13429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04: No bootstrap required, opened a new log
I20260810 22:29:37.935825 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:37.922349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.1.221.190:42475)
0810 22:29:37.922766 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:37.922788 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:37.922834 (+ 46us) server_negotiation.cc:407] Connection header received
0810 22:29:37.923685 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:37.923709 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:37.924122 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:37.924468 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:37.925154 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:37.926635 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:37.927518 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:37.928282 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:37.928450 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:37.931164 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:37.931171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:37.931193 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:37.931218 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:37.932957 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:37.933470 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:37.933475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:37.933480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:37.933646 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:37.933933 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:37.933936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:37.933937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:37.934302 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:37.934443 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:37.934820 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 22:29:37.935099 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":99,"threads_started":1}
I20260810 22:29:37.936653 13429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133e6ea470714266ae55aed838995d04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42475 } }
I20260810 22:29:37.936914 13429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:37.936981 13429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133e6ea470714266ae55aed838995d04, State: Initialized, Role: FOLLOWER
I20260810 22:29:37.937461 13429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133e6ea470714266ae55aed838995d04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42475 } }
I20260810 22:29:37.937634 13429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:29:37.937705 13429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:29:37.937798 13429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:37.938769 13429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133e6ea470714266ae55aed838995d04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42475 } }
I20260810 22:29:37.939052 13429 leader_election.cc:302] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 133e6ea470714266ae55aed838995d04; no voters:
I20260810 22:29:37.939410 13429 leader_election.cc:288] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:29:37.939524 13436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:37.939790 13436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [term 1 LEADER]: Becoming Leader. State: Replica: 133e6ea470714266ae55aed838995d04, State: Running, Role: LEADER
I20260810 22:29:37.940099 13436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133e6ea470714266ae55aed838995d04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42475 } }
I20260810 22:29:37.940670 13429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:29:37.941145 13436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133e6ea470714266ae55aed838995d04. Latest consensus state: current_term: 1 leader_uuid: "133e6ea470714266ae55aed838995d04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133e6ea470714266ae55aed838995d04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42475 } } }
I20260810 22:29:37.941269 13436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:37.942390 13442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:29:37.942417 13438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "133e6ea470714266ae55aed838995d04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133e6ea470714266ae55aed838995d04" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42475 } } }
I20260810 22:29:37.942624 13438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04 [sys.catalog]: This master's current role is: LEADER
I20260810 22:29:37.945604 13442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:29:37.951056 13442 catalog_manager.cc:1347] Generated new cluster ID: ece752d577c5445d94c36500337d00dc
I20260810 22:29:37.951128 13442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:29:37.958446 13442 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:29:37.959391 13442 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:29:37.963960 13442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04: Generated new TSK 0
I20260810 22:29:37.964623 13442 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:29:38.000911 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42475
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:38.133416 13460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:38.133728 13460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:38.133808 13460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:38.137833 13460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:38.137916 13460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:38.138001 13460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:29:38.143661 13460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:38.144699 13460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:38.145745 13460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:38.155683 13466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:38.155750 13470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:38.155782 13460 server_base.cc:1034] running on GCE node
W20260810 22:29:38.155952 13467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:38.156388 13460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:38.156985 13460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:38.158172 13460 hybrid_clock.cc:648] HybridClock initialized: now 1786400978158141 us; error 43 us; skew 500 ppm
Aug 10 22:29:38 dist-test-slave-njxd krb5kdc[13337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:38.160882 13460 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:29:38.161844 13460 webserver.cc:460] Webserver started at http://127.1.221.129:44655/ using document root <none> and password file <none>
I20260810 22:29:38.162427 13460 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:38.162521 13460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:38.162751 13460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:38.164727 13460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "12ef8d5f49e946fca67a972cafcf43fc"
format_stamp: "Formatted at 2026-08-10 22:29:38 on dist-test-slave-njxd"
server_key: "bf57c61526424e7eb25096888d810fbb"
server_key_iv: "7f67ad0e4585c67380cf38670baac095"
server_key_version: "encryptionkey@0"
I20260810 22:29:38.165306 13460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "12ef8d5f49e946fca67a972cafcf43fc"
format_stamp: "Formatted at 2026-08-10 22:29:38 on dist-test-slave-njxd"
server_key: "bf57c61526424e7eb25096888d810fbb"
server_key_iv: "7f67ad0e4585c67380cf38670baac095"
server_key_version: "encryptionkey@0"
I20260810 22:29:38.169261 13460 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:29:38.171754 13477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:38.172942 13460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:29:38.173077 13460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "12ef8d5f49e946fca67a972cafcf43fc"
format_stamp: "Formatted at 2026-08-10 22:29:38 on dist-test-slave-njxd"
server_key: "bf57c61526424e7eb25096888d810fbb"
server_key_iv: "7f67ad0e4585c67380cf38670baac095"
server_key_version: "encryptionkey@0"
I20260810 22:29:38.173187 13460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:38.187448 13460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:38.190516 13460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:38.190727 13460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:38.191341 13460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:38.192139 13460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:38.192214 13460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:38.192288 13460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:38.192332 13460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:38.201395 13460 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:41017
I20260810 22:29:38.201442 13609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:41017 every 8 connection(s)
I20260810 22:29:38.202318 13460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:29:38.208324 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 13460
I20260810 22:29:38.208444 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:29:38.208774 1910 external_mini_cluster.cc:1428] Setting key 957dec3f0c686454987abca2a7ab2591
Aug 10 22:29:38 dist-test-slave-njxd krb5kdc[13337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:29:38.217816 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.204083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:56689 (local address 127.1.221.190:42475)
0810 22:29:38.204221 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:38.204224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:38.205028 (+ 804us) server_negotiation.cc:407] Connection header received
0810 22:29:38.206468 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:38.206471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:38.206534 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:38.206615 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:38.207809 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.208600 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.209911 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.210277 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.210386 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.213678 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:38.213682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:38.213684 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:38.213708 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:38.215344 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:38.215913 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:38.215916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:38.215918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:38.216014 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:38.216364 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:38.216367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:38.216369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:38.216612 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:38.216710 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:38.217499 (+ 789us) server_negotiation.cc:299] Negotiation successful
0810 22:29:38.217676 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:29:38.219046 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.204416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:42475 (local address 127.1.221.129:56689)
0810 22:29:38.204864 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:38.204909 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:38.206235 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:38.206799 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:38.206807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:38.207166 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:38.207645 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.207659 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.208763 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.208766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.209791 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.209801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.210452 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.210457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.210673 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.211321 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:38.211349 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:38.213475 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:38.215488 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:38.215494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:38.215510 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:38.215785 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:38.216108 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:38.216112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:38.216114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:38.216260 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:38.216748 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:38.216758 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:38.217166 (+ 408us) client_negotiation.cc:769] Sending connection context
0810 22:29:38.217423 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 22:29:38.217846 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260810 22:29:38.220463 13610 heartbeater.cc:347] Connected to a master server at 127.1.221.190:42475
I20260810 22:29:38.220712 13610 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:38.221370 13610 heartbeater.cc:511] Master 127.1.221.190:42475 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260810 22:29:38.223038 13389 ts_manager.cc:194] Registered new tserver with Master: 12ef8d5f49e946fca67a972cafcf43fc (127.1.221.129:41017)
I20260810 22:29:38.224232 13389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:56689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:29:38.240494 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42475
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:38.355703 13615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:38.355947 13615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:38.355989 13615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:38.360010 13615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:38.360065 13615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:38.360124 13615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:29:38.365254 13615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:38.366326 13615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:38.367363 13615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:38.377086 13625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:38.377125 13615 server_base.cc:1034] running on GCE node
W20260810 22:29:38.377085 13622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:38.377087 13623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:38.377871 13615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:38.378445 13615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:38.379607 13615 hybrid_clock.cc:648] HybridClock initialized: now 1786400978379594 us; error 23 us; skew 500 ppm
Aug 10 22:29:38 dist-test-slave-njxd krb5kdc[13337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:38.382310 13615 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:29:38.383193 13615 webserver.cc:460] Webserver started at http://127.1.221.130:43649/ using document root <none> and password file <none>
I20260810 22:29:38.383780 13615 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:38.383873 13615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:38.384105 13615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:38.386185 13615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "6c1d11f39ceb4f4c90c5444305a300c9"
format_stamp: "Formatted at 2026-08-10 22:29:38 on dist-test-slave-njxd"
server_key: "26a0b1e9514cc6675100b6e099719436"
server_key_iv: "a2e7ae3c9eabb040afa7bf9c5928e009"
server_key_version: "encryptionkey@0"
I20260810 22:29:38.386749 13615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "6c1d11f39ceb4f4c90c5444305a300c9"
format_stamp: "Formatted at 2026-08-10 22:29:38 on dist-test-slave-njxd"
server_key: "26a0b1e9514cc6675100b6e099719436"
server_key_iv: "a2e7ae3c9eabb040afa7bf9c5928e009"
server_key_version: "encryptionkey@0"
I20260810 22:29:38.390673 13615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:29:38.393015 13635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:38.394264 13615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:29:38.394397 13615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "6c1d11f39ceb4f4c90c5444305a300c9"
format_stamp: "Formatted at 2026-08-10 22:29:38 on dist-test-slave-njxd"
server_key: "26a0b1e9514cc6675100b6e099719436"
server_key_iv: "a2e7ae3c9eabb040afa7bf9c5928e009"
server_key_version: "encryptionkey@0"
I20260810 22:29:38.394506 13615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:38.406386 13615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:38.409101 13615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:38.409309 13615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:38.409936 13615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:38.410727 13615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:38.410796 13615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:38.410866 13615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:38.410918 13615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:38.419963 13615 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:41837
I20260810 22:29:38.419992 13767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:41837 every 8 connection(s)
I20260810 22:29:38.420858 13615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:29:38.428169 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 13615
I20260810 22:29:38.428324 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:29:38.428647 1910 external_mini_cluster.cc:1428] Setting key 0c8a9bc37b66ec4d7b2a9ccab35bbe1c
Aug 10 22:29:38 dist-test-slave-njxd krb5kdc[13337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:29:38.435828 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.422783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:37071 (local address 127.1.221.190:42475)
0810 22:29:38.422930 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:38.422935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:38.423900 (+ 965us) server_negotiation.cc:407] Connection header received
0810 22:29:38.425216 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:38.425219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:38.425267 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:38.425341 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:38.426569 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.427112 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.428263 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.428612 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.428744 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.431657 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:38.431664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:38.431670 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:38.431704 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:38.433222 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:38.433933 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:38.433937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:38.433938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:38.434028 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:38.434482 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:38.434485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:38.434486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:38.434710 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:38.434789 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:38.435559 (+ 770us) server_negotiation.cc:299] Negotiation successful
0810 22:29:38.435684 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 22:29:38.437101 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.423222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:42475 (local address 127.1.221.130:37071)
0810 22:29:38.423744 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:38.423791 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:38.425021 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:38.425479 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:38.425486 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:38.425916 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:38.426426 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.426442 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.427228 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.427230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.428131 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.428141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.428817 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.428821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.428973 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.429694 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:38.429715 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:38.431482 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:38.433343 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:38.433353 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:38.433371 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:38.433807 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:38.434140 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:38.434146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:38.434151 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:38.434350 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:38.434833 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:38.434843 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:38.435212 (+ 369us) client_negotiation.cc:769] Sending connection context
0810 22:29:38.435450 (+ 238us) client_negotiation.cc:240] Negotiation successful
0810 22:29:38.435878 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":110,"threads_started":1}
I20260810 22:29:38.438488 13768 heartbeater.cc:347] Connected to a master server at 127.1.221.190:42475
I20260810 22:29:38.438759 13768 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:38.439266 13768 heartbeater.cc:511] Master 127.1.221.190:42475 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260810 22:29:38.440389 13389 ts_manager.cc:194] Registered new tserver with Master: 6c1d11f39ceb4f4c90c5444305a300c9 (127.1.221.130:41837)
I20260810 22:29:38.441147 13389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:37071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:29:38.460872 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42475
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:29:38.575393 13774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:29:38.575644 13774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:29:38.575719 13774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:29:38.579879 13774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:29:38.579936 13774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:29:38.580037 13774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:29:38.585316 13774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:42475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:29:38.586380 13774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:29:38.587481 13774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:29:38.597055 13782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:38.597216 13774 server_base.cc:1034] running on GCE node
W20260810 22:29:38.597059 13785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:29:38.597033 13780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:29:38.597891 13774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:29:38.598596 13774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:29:38.599751 13774 hybrid_clock.cc:648] HybridClock initialized: now 1786400978599732 us; error 32 us; skew 500 ppm
Aug 10 22:29:38 dist-test-slave-njxd krb5kdc[13337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:29:38.606949 13774 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:29:38.607913 13774 webserver.cc:460] Webserver started at http://127.1.221.131:37925/ using document root <none> and password file <none>
I20260810 22:29:38.608498 13774 fs_manager.cc:359] Metadata directory not provided
I20260810 22:29:38.608593 13774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:29:38.608834 13774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:29:38.610937 13774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "80ed6f239eac4feea8a79f80908ea4ce"
format_stamp: "Formatted at 2026-08-10 22:29:38 on dist-test-slave-njxd"
server_key: "161ec3b5995b8113ec71338c343a0c61"
server_key_iv: "e54836dd162cbe49d18b729da1330a6d"
server_key_version: "encryptionkey@0"
I20260810 22:29:38.611507 13774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "80ed6f239eac4feea8a79f80908ea4ce"
format_stamp: "Formatted at 2026-08-10 22:29:38 on dist-test-slave-njxd"
server_key: "161ec3b5995b8113ec71338c343a0c61"
server_key_iv: "e54836dd162cbe49d18b729da1330a6d"
server_key_version: "encryptionkey@0"
I20260810 22:29:38.615211 13774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:29:38.617491 13792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:38.618741 13774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:29:38.618892 13774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "80ed6f239eac4feea8a79f80908ea4ce"
format_stamp: "Formatted at 2026-08-10 22:29:38 on dist-test-slave-njxd"
server_key: "161ec3b5995b8113ec71338c343a0c61"
server_key_iv: "e54836dd162cbe49d18b729da1330a6d"
server_key_version: "encryptionkey@0"
I20260810 22:29:38.619004 13774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:29:38.632357 13774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:29:38.635317 13774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:29:38.635527 13774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:29:38.636137 13774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:29:38.636979 13774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:29:38.637051 13774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:38.637130 13774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:29:38.637173 13774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:29:38.646160 13774 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:46029
I20260810 22:29:38.646353 13915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:46029 every 8 connection(s)
I20260810 22:29:38.647115 13774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:29:38.648089 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 13774
I20260810 22:29:38.648231 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:29:38.648566 1910 external_mini_cluster.cc:1428] Setting key 3c34e99fb371ab39c65b19a61e10264b
Aug 10 22:29:38 dist-test-slave-njxd krb5kdc[13337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:29:38.661118 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.649200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:40229 (local address 127.1.221.190:42475)
0810 22:29:38.649354 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:38.649357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:38.650329 (+ 972us) server_negotiation.cc:407] Connection header received
0810 22:29:38.651955 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:38.651960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:38.652021 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:38.652115 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:38.653365 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.654083 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.655239 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.655448 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.655525 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.658125 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:38.658128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:38.658130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:38.658153 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:38.659389 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:38.659846 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:38.659849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:38.659850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:38.659920 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:38.660216 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:38.660218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:38.660219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:38.660382 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:38.660456 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:38.660899 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 22:29:38.660996 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:29:38.662034 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.649686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:42475 (local address 127.1.221.131:40229)
0810 22:29:38.650168 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:38.650212 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:38.651735 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:38.652332 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:38.652340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:38.652720 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:29:38.653228 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.653241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.654203 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.654206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.655120 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.655138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.655580 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.655582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.655736 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.656373 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:29:38.656390 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:29:38.657964 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:29:38.659495 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:38.659500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:38.659512 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:38.659763 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:38.659998 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:29:38.660001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:29:38.660002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:29:38.660137 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:29:38.660468 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:29:38.660473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:29:38.660701 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 22:29:38.660853 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 22:29:38.661097 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":124,"threads_started":1}
I20260810 22:29:38.663235 13916 heartbeater.cc:347] Connected to a master server at 127.1.221.190:42475
I20260810 22:29:38.663549 13916 heartbeater.cc:464] Registering TS with master...
I20260810 22:29:38.664124 13916 heartbeater.cc:511] Master 127.1.221.190:42475 requested a full tablet report, sending...
I20260810 22:29:38.665108 13384 ts_manager.cc:194] Registered new tserver with Master: 80ed6f239eac4feea8a79f80908ea4ce (127.1.221.131:46029)
I20260810 22:29:38.665762 13384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:40229
I20260810 22:29:38.675513 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:29:38.685015 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.677533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43610 (local address 127.1.221.190:42475)
0810 22:29:38.677718 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:38.677722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:38.677897 (+ 175us) server_negotiation.cc:407] Connection header received
0810 22:29:38.678046 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:38.678049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:38.678101 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:38.678190 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:29:38.678932 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.679441 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.680296 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.680519 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.680624 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.682105 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:29:38.682108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:29:38.682111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:29:38.682144 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:29:38.683374 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:38.683787 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:38.683791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:38.683793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:38.683870 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:29:38.684132 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:29:38.684137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:29:38.684141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:29:38.684345 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:29:38.684483 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:38.684742 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 22:29:38.684858 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 22:29:38.687764 13384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:29:38.690992 13384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:29:38.705421 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.699090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:46029 (local address 127.0.0.1:54882)
0810 22:29:38.699421 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:38.699466 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:38.699582 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:38.700740 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:38.700746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:38.700766 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:38.700898 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.700905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.702581 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.702584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.703791 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.703798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.704883 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.704887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.704994 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.705033 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 22:29:38.705155 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 22:29:38.705270 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":84,"thread_start_us":109,"threads_started":1}
I20260810 22:29:38.705471 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.699208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.1.221.131:46029)
0810 22:29:38.699899 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:38.699912 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:38.699936 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:29:38.700013 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:38.700019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:38.700247 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:38.701053 (+ 806us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:38.701214 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.702410 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.703924 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.704765 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.704871 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.705001 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:38.705215 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 22:29:38.705324 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":145,"threads_started":1}
I20260810 22:29:38.706676 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.699203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41017 (local address 127.0.0.1:42610)
0810 22:29:38.700470 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:38.700481 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:38.700591 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:38.701027 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:38.701030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:38.701044 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:38.701148 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.701154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.702811 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.702816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.703779 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.703791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.706293 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.706298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.706387 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.706410 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:29:38.706488 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 22:29:38.706551 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"mutex_wait_us":39,"thread_start_us":131,"threads_started":1}
I20260810 22:29:38.706800 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.699906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.1.221.129:41017)
0810 22:29:38.700323 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:38.700340 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:38.700574 (+ 234us) server_negotiation.cc:407] Connection header received
0810 22:29:38.700707 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:38.700713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:38.700896 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:38.701052 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:38.701245 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.702696 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.705057 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.705957 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.706066 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.706216 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:38.706544 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 22:29:38.706652 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":98,"threads_started":1}
I20260810 22:29:38.706895 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.699953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:41837 (local address 127.0.0.1:36830)
0810 22:29:38.700760 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:38.700782 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:38.700901 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:38.701394 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:38.701397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:38.701414 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:38.701515 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.701521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.702482 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.702486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.704707 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.704720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.706444 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.706446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.706564 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.706586 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:29:38.706680 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 22:29:38.706750 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":52,"thread_start_us":86,"threads_started":1}
I20260810 22:29:38.707271 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.700356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.1.221.130:41837)
0810 22:29:38.700699 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:38.700711 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:38.700881 (+ 170us) server_negotiation.cc:407] Connection header received
0810 22:29:38.701044 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:38.701050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:38.701245 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:38.701419 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:38.701630 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.702352 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.704871 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.706153 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.706255 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.706391 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:38.706987 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 22:29:38.707086 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":143,"threads_started":1}
I20260810 22:29:38.708978 13848 tablet_service.cc:1467] Processing CreateTablet for tablet e1c6e2d095ce41f3aa49b9e25b9810e0 (DEFAULT_TABLE table=test-table [id=3fec5a55380a4cdc918b71e020d2e9cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:38.709450 13532 tablet_service.cc:1467] Processing CreateTablet for tablet e1c6e2d095ce41f3aa49b9e25b9810e0 (DEFAULT_TABLE table=test-table [id=3fec5a55380a4cdc918b71e020d2e9cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:38.710101 13848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c6e2d095ce41f3aa49b9e25b9810e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:38.710516 13532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c6e2d095ce41f3aa49b9e25b9810e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:38.712064 13693 tablet_service.cc:1467] Processing CreateTablet for tablet e1c6e2d095ce41f3aa49b9e25b9810e0 (DEFAULT_TABLE table=test-table [id=3fec5a55380a4cdc918b71e020d2e9cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:29:38.713119 13693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c6e2d095ce41f3aa49b9e25b9810e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:29:38.716630 13936 tablet_bootstrap.cc:492] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc: Bootstrap starting.
I20260810 22:29:38.718186 13937 tablet_bootstrap.cc:492] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce: Bootstrap starting.
I20260810 22:29:38.718986 13936 tablet_bootstrap.cc:654] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:38.720252 13936 log.cc:824] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:38.720444 13938 tablet_bootstrap.cc:492] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9: Bootstrap starting.
I20260810 22:29:38.720888 13937 tablet_bootstrap.cc:654] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:38.721642 13937 log.cc:824] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:38.722932 13938 tablet_bootstrap.cc:654] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9: Neither blocks nor log segments found. Creating new log.
I20260810 22:29:38.723495 13937 tablet_bootstrap.cc:492] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce: No bootstrap required, opened a new log
I20260810 22:29:38.723574 13936 tablet_bootstrap.cc:492] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc: No bootstrap required, opened a new log
I20260810 22:29:38.723685 13937 ts_tablet_manager.cc:1397] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:29:38.723757 13936 ts_tablet_manager.cc:1397] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 22:29:38.723762 13938 log.cc:824] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9: Log is configured to *not* fsync() on all Append() calls
I20260810 22:29:38.725809 13938 tablet_bootstrap.cc:492] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9: No bootstrap required, opened a new log
I20260810 22:29:38.725975 13938 ts_tablet_manager.cc:1397] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:29:38.726886 13937 raft_consensus.cc:348] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ed6f239eac4feea8a79f80908ea4ce" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 46029 } } peers { permanent_uuid: "12ef8d5f49e946fca67a972cafcf43fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41017 } } peers { permanent_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41837 } }
I20260810 22:29:38.727114 13937 raft_consensus.cc:374] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:38.727195 13937 raft_consensus.cc:729] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ed6f239eac4feea8a79f80908ea4ce, State: Initialized, Role: FOLLOWER
I20260810 22:29:38.727646 13937 consensus_queue.cc:260] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ed6f239eac4feea8a79f80908ea4ce" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 46029 } } peers { permanent_uuid: "12ef8d5f49e946fca67a972cafcf43fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41017 } } peers { permanent_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41837 } }
I20260810 22:29:38.727530 13936 raft_consensus.cc:348] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ed6f239eac4feea8a79f80908ea4ce" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 46029 } } peers { permanent_uuid: "12ef8d5f49e946fca67a972cafcf43fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41017 } } peers { permanent_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41837 } }
I20260810 22:29:38.727787 13936 raft_consensus.cc:374] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:38.727883 13936 raft_consensus.cc:729] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ef8d5f49e946fca67a972cafcf43fc, State: Initialized, Role: FOLLOWER
I20260810 22:29:38.728291 13916 heartbeater.cc:503] Master 127.1.221.190:42475 was elected leader, sending a full tablet report...
I20260810 22:29:38.728300 13936 consensus_queue.cc:260] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ed6f239eac4feea8a79f80908ea4ce" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 46029 } } peers { permanent_uuid: "12ef8d5f49e946fca67a972cafcf43fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41017 } } peers { permanent_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41837 } }
I20260810 22:29:38.728605 13937 ts_tablet_manager.cc:1428] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:29:38.729197 13936 ts_tablet_manager.cc:1428] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 22:29:38.729581 13610 heartbeater.cc:503] Master 127.1.221.190:42475 was elected leader, sending a full tablet report...
I20260810 22:29:38.729624 13938 raft_consensus.cc:348] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ed6f239eac4feea8a79f80908ea4ce" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 46029 } } peers { permanent_uuid: "12ef8d5f49e946fca67a972cafcf43fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41017 } } peers { permanent_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41837 } }
I20260810 22:29:38.729877 13938 raft_consensus.cc:374] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:29:38.729921 13938 raft_consensus.cc:729] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1d11f39ceb4f4c90c5444305a300c9, State: Initialized, Role: FOLLOWER
I20260810 22:29:38.730275 13938 consensus_queue.cc:260] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ed6f239eac4feea8a79f80908ea4ce" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 46029 } } peers { permanent_uuid: "12ef8d5f49e946fca67a972cafcf43fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41017 } } peers { permanent_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41837 } }
I20260810 22:29:38.731173 13938 ts_tablet_manager.cc:1428] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 22:29:38.731542 13768 heartbeater.cc:503] Master 127.1.221.190:42475 was elected leader, sending a full tablet report...
W20260810 22:29:38.898299 13917 tablet.cc:2369] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:29:38.922046 13769 tablet.cc:2369] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:29:38.953816 13611 tablet.cc:2369] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:29:38.977802 13944 raft_consensus.cc:482] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:29:38.978057 13944 raft_consensus.cc:504] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ed6f239eac4feea8a79f80908ea4ce" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 46029 } } peers { permanent_uuid: "12ef8d5f49e946fca67a972cafcf43fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41017 } } peers { permanent_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41837 } }
I20260810 22:29:38.979259 13944 leader_election.cc:288] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80ed6f239eac4feea8a79f80908ea4ce (127.1.221.131:46029), 12ef8d5f49e946fca67a972cafcf43fc (127.1.221.129:41017)
I20260810 22:29:38.983266 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.979832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:36431 (local address 127.1.221.129:41017)
0810 22:29:38.979957 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:38.979963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:38.980070 (+ 107us) server_negotiation.cc:407] Connection header received
0810 22:29:38.980233 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:38.980236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:38.980291 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:38.980392 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:38.980667 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.981194 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.982319 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.982711 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.982840 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.982888 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:38.983072 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 22:29:38.983141 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:29:38.983266 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.979679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:44705 (local address 127.1.221.131:46029)
0810 22:29:38.979888 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:38.979894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:38.979913 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:29:38.980091 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:38.980097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:38.980143 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:38.980224 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:29:38.980528 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.981094 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.982274 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.982667 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.982805 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.982861 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:38.983047 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 22:29:38.983132 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260810 22:29:38.983290 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.979510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41017 (local address 127.1.221.130:36431)
0810 22:29:38.979956 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:38.979971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:38.980114 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:38.980403 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:38.980406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:38.980423 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:38.980550 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.980555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.981312 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.981316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.982194 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.982203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.982833 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.982836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.982932 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.982959 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:29:38.983040 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 22:29:38.983138 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":53,"thread_start_us":80,"threads_started":1}
I20260810 22:29:38.983304 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:38.979503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:46029 (local address 127.1.221.130:44705)
0810 22:29:38.979807 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:38.979825 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:29:38.979954 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:29:38.980248 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:29:38.980251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:29:38.980275 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:29:38.980416 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.980422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.981209 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.981214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.982135 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:29:38.982145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:29:38.982791 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:29:38.982795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:29:38.982898 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:38.982934 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 22:29:38.983017 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 22:29:38.983143 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":61,"thread_start_us":103,"threads_started":1}
I20260810 22:29:38.983954 13554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c6e2d095ce41f3aa49b9e25b9810e0" candidate_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ef8d5f49e946fca67a972cafcf43fc" is_pre_election: true
I20260810 22:29:38.984303 13554 raft_consensus.cc:2454] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1d11f39ceb4f4c90c5444305a300c9 in term 0.
I20260810 22:29:38.984408 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c6e2d095ce41f3aa49b9e25b9810e0" candidate_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ed6f239eac4feea8a79f80908ea4ce" is_pre_election: true
I20260810 22:29:38.984719 13869 raft_consensus.cc:2454] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c1d11f39ceb4f4c90c5444305a300c9 in term 0.
I20260810 22:29:38.984781 13640 leader_election.cc:302] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12ef8d5f49e946fca67a972cafcf43fc, 6c1d11f39ceb4f4c90c5444305a300c9; no voters:
I20260810 22:29:38.985097 13944 raft_consensus.cc:2781] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:29:38.985211 13944 raft_consensus.cc:482] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:29:38.985275 13944 raft_consensus.cc:3037] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:38.986351 13944 raft_consensus.cc:504] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ed6f239eac4feea8a79f80908ea4ce" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 46029 } } peers { permanent_uuid: "12ef8d5f49e946fca67a972cafcf43fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41017 } } peers { permanent_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41837 } }
I20260810 22:29:38.986734 13944 leader_election.cc:288] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [CANDIDATE]: Term 1 election: Requested vote from peers 80ed6f239eac4feea8a79f80908ea4ce (127.1.221.131:46029), 12ef8d5f49e946fca67a972cafcf43fc (127.1.221.129:41017)
I20260810 22:29:38.987099 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c6e2d095ce41f3aa49b9e25b9810e0" candidate_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80ed6f239eac4feea8a79f80908ea4ce"
I20260810 22:29:38.987109 13554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c6e2d095ce41f3aa49b9e25b9810e0" candidate_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ef8d5f49e946fca67a972cafcf43fc"
I20260810 22:29:38.987228 13869 raft_consensus.cc:3037] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:38.987228 13554 raft_consensus.cc:3037] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:29:38.988579 13869 raft_consensus.cc:2454] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1d11f39ceb4f4c90c5444305a300c9 in term 1.
I20260810 22:29:38.988586 13554 raft_consensus.cc:2454] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c1d11f39ceb4f4c90c5444305a300c9 in term 1.
I20260810 22:29:38.988896 13637 leader_election.cc:302] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c1d11f39ceb4f4c90c5444305a300c9, 80ed6f239eac4feea8a79f80908ea4ce; no voters:
I20260810 22:29:38.989089 13944 raft_consensus.cc:2781] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:29:38.989351 13944 raft_consensus.cc:686] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [term 1 LEADER]: Becoming Leader. State: Replica: 6c1d11f39ceb4f4c90c5444305a300c9, State: Running, Role: LEADER
I20260810 22:29:38.989748 13944 consensus_queue.cc:237] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ed6f239eac4feea8a79f80908ea4ce" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 46029 } } peers { permanent_uuid: "12ef8d5f49e946fca67a972cafcf43fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41017 } } peers { permanent_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41837 } }
I20260810 22:29:38.993039 13384 catalog_manager.cc:5495] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c1d11f39ceb4f4c90c5444305a300c9 (127.1.221.130). New cstate: current_term: 1 leader_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ed6f239eac4feea8a79f80908ea4ce" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 46029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12ef8d5f49e946fca67a972cafcf43fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c1d11f39ceb4f4c90c5444305a300c9" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41837 } health_report { overall_health: HEALTHY } } }
I20260810 22:29:39.070894 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:39.067143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.1.221.130:41837)
0810 22:29:39.067287 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 22:29:39.067292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:29:39.067308 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:29:39.067437 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:29:39.067440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:29:39.067501 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:29:39.067622 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:29:39.067875 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:39.068387 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:39.069277 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:29:39.069560 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:29:39.069697 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:29:39.070030 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:29:39.070480 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:29:39.070622 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 22:29:39.070680 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 22:29:39.070738 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:29:39.078878 13554 raft_consensus.cc:1261] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc [term 1 FOLLOWER]: Refusing update from remote peer 6c1d11f39ceb4f4c90c5444305a300c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:39.078879 13869 raft_consensus.cc:1261] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce [term 1 FOLLOWER]: Refusing update from remote peer 6c1d11f39ceb4f4c90c5444305a300c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:29:39.079668 13944 consensus_queue.cc:1035] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80ed6f239eac4feea8a79f80908ea4ce" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 46029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:39.079907 13947 consensus_queue.cc:1035] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ef8d5f49e946fca67a972cafcf43fc" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:29:39.087710 13951 mvcc.cc:205] Tried to move back new op lower bound from 7317098410298261504 to 7317098409944600576. Current Snapshot: MvccSnapshot[applied={T|T < 7317098410298261504}]
I20260810 22:29:39.088161 13952 mvcc.cc:205] Tried to move back new op lower bound from 7317098410298261504 to 7317098409944600576. Current Snapshot: MvccSnapshot[applied={T|T < 7317098410298261504}]
I20260810 22:29:39.090113 13954 mvcc.cc:205] Tried to move back new op lower bound from 7317098410298261504 to 7317098409944600576. Current Snapshot: MvccSnapshot[applied={T|T < 7317098410298261504}]
I20260810 22:29:39.096902 13384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:39.097088 13384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:29:39.099587 13384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 133e6ea470714266ae55aed838995d04: Sending DeleteTablet for 3 replicas of tablet e1c6e2d095ce41f3aa49b9e25b9810e0
I20260810 22:29:39.100512 13848 tablet_service.cc:1514] Processing DeleteTablet for tablet e1c6e2d095ce41f3aa49b9e25b9810e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:39 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:54882
I20260810 22:29:39.100545 13532 tablet_service.cc:1514] Processing DeleteTablet for tablet e1c6e2d095ce41f3aa49b9e25b9810e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:39 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:42610
I20260810 22:29:39.101014 13965 tablet_replica.cc:330] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce: stopping tablet replica
I20260810 22:29:39.101188 13693 tablet_service.cc:1514] Processing DeleteTablet for tablet e1c6e2d095ce41f3aa49b9e25b9810e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:29:39 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:36830
I20260810 22:29:39.101403 13965 raft_consensus.cc:2229] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:39.101516 13966 tablet_replica.cc:330] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc: stopping tablet replica
I20260810 22:29:39.101764 13965 raft_consensus.cc:2258] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:39.101912 13966 raft_consensus.cc:2229] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 12ef8d5f49e946fca67a972cafcf43fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:29:39.102226 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 13460
I20260810 22:29:39.103569 13967 tablet_replica.cc:330] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9: stopping tablet replica
I20260810 22:29:39.103854 13967 raft_consensus.cc:2229] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:29:39.104187 13967 raft_consensus.cc:2258] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:29:39.106279 13967 ts_tablet_manager.cc:1905] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:39.107102 13965 ts_tablet_manager.cc:1905] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:29:39.109236 13967 ts_tablet_manager.cc:1918] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:39.109383 13967 log.cc:1196] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/e1c6e2d095ce41f3aa49b9e25b9810e0
I20260810 22:29:39.109778 13967 ts_tablet_manager.cc:1939] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 6c1d11f39ceb4f4c90c5444305a300c9: Deleting consensus metadata
I20260810 22:29:39.110076 13965 ts_tablet_manager.cc:1918] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:29:39.110257 13965 log.cc:1196] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/e1c6e2d095ce41f3aa49b9e25b9810e0
I20260810 22:29:39.110668 13965 ts_tablet_manager.cc:1939] T e1c6e2d095ce41f3aa49b9e25b9810e0 P 80ed6f239eac4feea8a79f80908ea4ce: Deleting consensus metadata
I20260810 22:29:39.111171 13376 catalog_manager.cc:4841] TS 6c1d11f39ceb4f4c90c5444305a300c9 (127.1.221.130:41837): tablet e1c6e2d095ce41f3aa49b9e25b9810e0 (table test-table [id=3fec5a55380a4cdc918b71e020d2e9cc]) successfully deleted
I20260810 22:29:39.111914 13373 catalog_manager.cc:4841] TS 80ed6f239eac4feea8a79f80908ea4ce (127.1.221.131:46029): tablet e1c6e2d095ce41f3aa49b9e25b9810e0 (table test-table [id=3fec5a55380a4cdc918b71e020d2e9cc]) successfully deleted
W20260810 22:29:39.115738 13376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:41017 (error 108)
I20260810 22:29:39.115816 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 13615
I20260810 22:29:39.119989 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:29:39.119624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41017 (local address 127.0.0.1:42614)
0810 22:29:39.119783 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 22:29:39.119871 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:41017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 22:29:39.120312 13376 catalog_manager.cc:4584] TS 12ef8d5f49e946fca67a972cafcf43fc (127.1.221.129:41017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1c6e2d095ce41f3aa49b9e25b9810e0: Network error: Client connection negotiation failed: client connection to 127.1.221.129:41017: connect: Connection refused (error 111)
I20260810 22:29:39.125957 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 13774
I20260810 22:29:39.133262 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 13354
2026-08-10T22:29:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2641 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:29:39 dist-test-slave-njxd krb5kdc[13971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:29:39 dist-test-slave-njxd krb5kdc[13971](info): set up 2 sockets
Aug 10 22:29:39 dist-test-slave-njxd krb5kdc[13971](info): commencing operation
krb5kdc: starting...
W20260810 22:29:40.242792 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:29:40 dist-test-slave-njxd krb5kdc[13971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786400980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:29:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:29:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/rangeradmin_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/rangeradmin_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/rangerlookup_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/rangerlookup_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
I20260810 22:29:40.420605 1910 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYvFf7I/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/postgres -l logfile start
2026-08-10 22:29:42.861 UTC [14004] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 22:29:42.861 UTC [14004] LOG: listening on IPv4 address "127.1.221.148", port 59171
2026-08-10 22:29:42.865 UTC [14004] LOG: listening on Unix socket "/tmp/.s.PGSQL.59171"
2026-08-10 22:29:42.900 UTC [14007] LOG: database system was shut down at 2026-08-10 22:29:41 UTC
2026-08-10 22:29:42.904 UTC [14004] LOG: database system is ready to accept connections
I20260810 22:29:43.895572 1910 mini_postgres.cc:96] Postgres bound to 59171
2026-08-10 22:29:43.902 UTC [14015] FATAL: database "slave" does not exist
127.1.221.148:59171 - accepting connections
I20260810 22:29:43.903764 1910 mini_ranger.cc:161] Starting Ranger...
I20260810 22:29:43.919934 1910 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 22:29:44.386140 1910 mini_ranger.cc:87] Created ranger Postgres database
I20260810 22:29:44.386230 1910 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-admin
2026-08-10 22:29:44,679 [I] DB FLAVOR :POSTGRES
2026-08-10 22:29:44,680 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 22:29:44,680 [I] Checking connection..
2026-08-10 22:29:44,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 22:29:44,926 [I] Checking connection passed.
2026-08-10 22:29:44,926 [I] --------- Verifying version history table ---------
2026-08-10 22:29:44,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:29:45,184 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 22:29:45,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:29:45,440 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 22:29:45,441 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 22:29:45,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 22:29:45.702 UTC [14087] WARNING: there is no transaction in progress
2026-08-10 22:29:45,709 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 22:29:45,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:29:45,976 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 22:29:45,976 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 22:29:45,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 22:29:46,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 22:29:46,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-njxd.c.gcp-upstream.internal','N') ;"
2026-08-10 22:29:46,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 22:29:47,006 [I] Table x_portal_user does not exist in database ranger
2026-08-10 22:29:47,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 22:29:47,263 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 22:29:47,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 22:29:47,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 22:29:47,760 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 22:29:47,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 22:29:48.727 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.743 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.760 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.776 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.791 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.807 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.861 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.867 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.880 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.890 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.900 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.912 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.922 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.928 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.935 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:48.941 UTC [14209] WARNING: there is no transaction in progress
2026-08-10 22:29:49,551 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 22:29:49,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 22:29:49,784 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 22:29:49,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 22:29:50,039 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 22:29:50,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 22:29:50,305 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 22:29:50,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 22:29:50,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 22:29:50,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-njxd.c.gcp-upstream.internal';"
2026-08-10 22:29:51,095 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 22:29:51,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 22:29:51,356 [I] DB_PATCHES have already been applied
I20260810 22:29:51.362231 1910 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYvFf7I/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 22:29:51.362319 1910 mini_ranger.cc:193] Using host: 127.1.221.148
I20260810 22:29:51.366783 1910 mini_ranger.cc:227] Ranger admin URL: http://127.1.221.148:38807
Aug 10, 2026 10:29:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 10:29:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 10:29:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 10:29:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 10:29:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 10:29:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 10:29:52 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 22:29:52.537184 1910 mini_ranger.cc:160] Time spent starting Ranger: real 8.633s user 0.009s sys 0.004s
Aug 10, 2026 10:30:13 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 10:30:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 10:30:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 10:30:13 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 10:30:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 10:30:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 10:30:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 10:30:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 10:30:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 10:30:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 10:30:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 10:30:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 10:30:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 10:30:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 10:30:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 10:30:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 10:30:13 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 10:30:14 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 22:30:14.607494 1910 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/rangerkms_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/rangerkms_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.221.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.221.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 22:30:14 dist-test-slave-njxd krb5kdc[13971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401014, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 22:30:14.698119 1910 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 22:30:14.710172 1910 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 22:30:15.562453 1910 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 22:30:15.562558 1910 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-kms
I20260810 22:30:15.562683 1910 mini_ranger_kms.cc:227] Using postgres at 127.1.221.148:59171
2026-08-10 22:30:15,810 [I] DB FLAVOR :POSTGRES
2026-08-10 22:30:15,810 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 22:30:15,810 [I] Checking connection
2026-08-10 22:30:15,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 22:30:16,065 [I] connection success
2026-08-10 22:30:16,065 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 22:30:16,065 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 22:30:16,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 22:30:16,300 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 22:30:16,300 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 22:30:16,300 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 22:30:16,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:59171/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 22:30:16,580 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 22:30:17.062749 1910 mini_ranger_kms.cc:310] Created kms service
I20260810 22:30:17.257332 1910 mini_ranger_kms.cc:326] Created kudu user
I20260810 22:30:17.302804 1910 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 22:30:17 dist-test-slave-njxd krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401014, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:30:17.682464 1910 mini_ranger_kms.cc:384] Added ranger policy
I20260810 22:30:17.682623 1910 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 22:30:17.682644 1910 mini_ranger_kms.cc:240] Using host: 127.1.221.148
I20260810 22:30:17.687453 1910 mini_ranger_kms.cc:276] Ranger KMS PID: 14416
I20260810 22:30:17.687599 1910 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.221.148:46593
Aug 10, 2026 10:30:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 10:30:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 10:30:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 10:30:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 10:30:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 10:30:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 22:30:20.131091 1910 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.433s user 0.011s sys 0.217s
I20260810 22:30:20.131222 1910 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 22:30:20.131364 1910 mini_ranger_kms.cc:401] 127.1.221.148:46593/kms/v1/keys
Aug 10 22:30:20 dist-test-slave-njxd krb5kdc[13971](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786401020, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.221.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 22:30:20 dist-test-slave-njxd krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401020, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.221.148@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
Aug 10 22:30:21 dist-test-slave-njxd krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401020, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.221.148@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
Aug 10 22:30:23 dist-test-slave-njxd krb5kdc[13971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:30:23.557092 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33015
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:33015
--ranger_config_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:46593/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:30:23.674592 14474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:30:23.674877 14474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:30:23.674919 14474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:30:23.679070 14474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:30:23.679139 14474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:30:23.679158 14474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:30:23.679175 14474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:30:23.679190 14474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:30:23.684908 14474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.221.148:46593/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:33015
--ranger_config_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:33015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:30:23.686128 14474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:30:23.687203 14474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:30:23.697289 14479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:30:23.697495 14482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:30:23.697537 14474 server_base.cc:1034] running on GCE node
W20260810 22:30:23.697297 14480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:30:23.698261 14474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:30:23.699481 14474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:30:23.700683 14474 hybrid_clock.cc:648] HybridClock initialized: now 1786401023700660 us; error 44 us; skew 500 ppm
Aug 10 22:30:23 dist-test-slave-njxd krb5kdc[13971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:30:23.703629 14474 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:30:23.704659 14474 webserver.cc:460] Webserver started at http://127.1.221.190:39629/ using document root <none> and password file <none>
I20260810 22:30:23.705199 14474 fs_manager.cc:359] Metadata directory not provided
I20260810 22:30:23.705261 14474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:30:23.705544 14474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:30:23 dist-test-slave-njxd krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:30:23.763942 14474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "e4e4481eac1d4c318fa5510b555db9f9"
format_stamp: "Formatted at 2026-08-10 22:30:23 on dist-test-slave-njxd"
server_key: "b9eb87345d081ee90c5ae559c0f1286c"
server_key_iv: "2a5499ad21e8e3c35856b31e8c672887"
server_key_version: "kuduclusterkey@0"
I20260810 22:30:23.764703 14474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "e4e4481eac1d4c318fa5510b555db9f9"
format_stamp: "Formatted at 2026-08-10 22:30:23 on dist-test-slave-njxd"
server_key: "b9eb87345d081ee90c5ae559c0f1286c"
server_key_iv: "2a5499ad21e8e3c35856b31e8c672887"
server_key_version: "kuduclusterkey@0"
I20260810 22:30:23.769009 14474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:30:23.788877 14489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:30:23.790473 14474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:30:23.790652 14474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "e4e4481eac1d4c318fa5510b555db9f9"
format_stamp: "Formatted at 2026-08-10 22:30:23 on dist-test-slave-njxd"
server_key: "b9eb87345d081ee90c5ae559c0f1286c"
server_key_iv: "2a5499ad21e8e3c35856b31e8c672887"
server_key_version: "kuduclusterkey@0"
I20260810 22:30:23.790802 14474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:30:23.815375 14474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:30:23.818881 14474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:30:23.819098 14474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:30:23.827246 14474 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:33015
I20260810 22:30:23.827325 14551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:33015 every 8 connection(s)
I20260810 22:30:23.828285 14474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:30:23.831246 14552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:30:23.836474 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 14474
I20260810 22:30:23.836612 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:30:23.836922 1910 external_mini_cluster.cc:1428] Setting key 93c1ad1e772234c32670cf73eadb0246
I20260810 22:30:23.837204 14552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9: Bootstrap starting.
I20260810 22:30:23.839818 14552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9: Neither blocks nor log segments found. Creating new log.
I20260810 22:30:23.840860 14552 log.cc:824] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9: Log is configured to *not* fsync() on all Append() calls
I20260810 22:30:23.843230 14552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9: No bootstrap required, opened a new log
Aug 10 22:30:23 dist-test-slave-njxd krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:30:23.847535 14552 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e4481eac1d4c318fa5510b555db9f9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33015 } }
I20260810 22:30:23.847767 14552 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:30:23.847827 14552 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e4481eac1d4c318fa5510b555db9f9, State: Initialized, Role: FOLLOWER
I20260810 22:30:23.848373 14552 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e4481eac1d4c318fa5510b555db9f9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33015 } }
I20260810 22:30:23.848539 14552 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:30:23.848608 14552 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:30:23.848721 14552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:30:23.849889 14552 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e4481eac1d4c318fa5510b555db9f9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33015 } }
I20260810 22:30:23.850279 14552 leader_election.cc:302] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4e4481eac1d4c318fa5510b555db9f9; no voters:
I20260810 22:30:23.850620 14552 leader_election.cc:288] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:30:23.850766 14557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:30:23.851001 14557 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [term 1 LEADER]: Becoming Leader. State: Replica: e4e4481eac1d4c318fa5510b555db9f9, State: Running, Role: LEADER
I20260810 22:30:23.851397 14557 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e4481eac1d4c318fa5510b555db9f9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33015 } }
I20260810 22:30:23.852007 14552 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:30:23.852805 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:23.838339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.1.221.190:33015)
0810 22:30:23.838765 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0810 22:30:23.838781 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:30:23.838816 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:30:23.839415 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:30:23.839431 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:30:23.839750 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:30:23.840111 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:30:23.841043 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:23.842303 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:23.843460 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:23.843914 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:23.844034 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:23.846927 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:30:23.846933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:30:23.846947 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:30:23.846974 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:30:23.849070 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:30:23.849486 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:30:23.849494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:30:23.849503 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:30:23.849675 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:30:23.849972 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:30:23.849975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:30:23.849978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:30:23.850495 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:30:23.850728 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 22:30:23.851442 (+ 714us) server_negotiation.cc:299] Negotiation successful
0810 22:30:23.851721 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":146,"threads_started":1}
I20260810 22:30:23.854388 14559 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4e4481eac1d4c318fa5510b555db9f9. Latest consensus state: current_term: 1 leader_uuid: "e4e4481eac1d4c318fa5510b555db9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e4481eac1d4c318fa5510b555db9f9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33015 } } }
I20260810 22:30:23.854501 14559 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [sys.catalog]: This master's current role is: LEADER
I20260810 22:30:23.854734 14552 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 22:30:23.854926 14574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:30:23.853998 14558 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4e4481eac1d4c318fa5510b555db9f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e4481eac1d4c318fa5510b555db9f9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 33015 } } }
I20260810 22:30:23.855504 14558 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9 [sys.catalog]: This master's current role is: LEADER
I20260810 22:30:23.861107 14574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:30:23.868948 14574 catalog_manager.cc:1347] Generated new cluster ID: f4b576d19e7046bc9bac47f4594604bc
I20260810 22:30:23.869050 14574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:30:23.890619 14574 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:30:23.891930 14574 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:30:23.897698 14574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9: Generated new TSK 0
I20260810 22:30:23.898377 14574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:30:24.613649 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:46593/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33015
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:30:24.762598 14602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:30:24.762984 14602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:30:24.763067 14602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:30:24.769085 14602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:30:24.769181 14602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:30:24.769270 14602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:30:24.777129 14602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.221.148:46593/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:30:24.778631 14602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:30:24.780018 14602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:30:24.791779 14610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:30:24.791774 14611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:30:24.791817 14613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:30:24.792368 14602 server_base.cc:1034] running on GCE node
I20260810 22:30:24.792765 14602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:30:24.793429 14602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:30:24.794607 14602 hybrid_clock.cc:648] HybridClock initialized: now 1786401024794573 us; error 50 us; skew 500 ppm
Aug 10 22:30:24 dist-test-slave-njxd krb5kdc[13971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:30:24.797712 14602 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:30:24.798590 14602 webserver.cc:460] Webserver started at http://127.1.221.129:34899/ using document root <none> and password file <none>
I20260810 22:30:24.799135 14602 fs_manager.cc:359] Metadata directory not provided
I20260810 22:30:24.799201 14602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:30:24.799410 14602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:30:24 dist-test-slave-njxd krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:30:24.815621 14602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "e1d23dc48365407383453b5dc8eb0635"
format_stamp: "Formatted at 2026-08-10 22:30:24 on dist-test-slave-njxd"
server_key: "3d989100f65c4a41574484639ad187ff"
server_key_iv: "bab87be7fe203a46eda147ca61e293eb"
server_key_version: "kuduclusterkey@0"
I20260810 22:30:24.816435 14602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "e1d23dc48365407383453b5dc8eb0635"
format_stamp: "Formatted at 2026-08-10 22:30:24 on dist-test-slave-njxd"
server_key: "3d989100f65c4a41574484639ad187ff"
server_key_iv: "bab87be7fe203a46eda147ca61e293eb"
server_key_version: "kuduclusterkey@0"
I20260810 22:30:24.822820 14602 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 22:30:24.843369 14620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:30:24.845124 14602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:30:24.845337 14602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "e1d23dc48365407383453b5dc8eb0635"
format_stamp: "Formatted at 2026-08-10 22:30:24 on dist-test-slave-njxd"
server_key: "3d989100f65c4a41574484639ad187ff"
server_key_iv: "bab87be7fe203a46eda147ca61e293eb"
server_key_version: "kuduclusterkey@0"
I20260810 22:30:24.845604 14602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:30:24.864296 14602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:30:24.867744 14602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:30:24.867965 14602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:30:24.868541 14602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:30:24.869359 14602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:30:24.869406 14602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:30:24.869446 14602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:30:24.869494 14602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:30:24.881278 14602 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:35549
I20260810 22:30:24.882568 14602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:30:24.883654 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 14602
I20260810 22:30:24.883821 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:30:24.884109 1910 external_mini_cluster.cc:1428] Setting key 17b2bb2adc76606b7d6eae49b0fbadd5
I20260810 22:30:24.892190 14747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:35549 every 8 connection(s)
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Aug 10 22:30:24 dist-test-slave-njxd krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:30:24.913829 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:24.896408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:50629 (local address 127.1.221.190:33015)
0810 22:30:24.899401 (+ 2993us) server_negotiation.cc:206] Beginning negotiation
0810 22:30:24.899412 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:30:24.899429 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:30:24.899478 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:30:24.899483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:30:24.899541 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:30:24.899647 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:30:24.901335 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:24.902058 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:24.903452 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:24.903675 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:24.903805 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:24.908632 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:30:24.908650 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:30:24.908657 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:30:24.908691 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:30:24.910866 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:30:24.911653 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:30:24.911659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:30:24.911662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:30:24.911768 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:30:24.912242 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:30:24.912248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:30:24.912252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:30:24.912543 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:30:24.912627 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 22:30:24.913425 (+ 798us) server_negotiation.cc:299] Negotiation successful
0810 22:30:24.913569 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2897,"thread_start_us":116,"threads_started":1}
I20260810 22:30:24.915782 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:24.894915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33015 (local address 127.1.221.129:50629)
0810 22:30:24.896925 (+ 2010us) negotiation.cc:107] Waiting for socket to connect
0810 22:30:24.896989 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 22:30:24.898934 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:30:24.900005 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:30:24.900017 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:30:24.900390 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:30:24.901139 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:24.901173 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:24.902204 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:24.902209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:24.903311 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:24.903320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:24.903941 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:24.903946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:24.904080 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:24.904778 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:30:24.904803 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:30:24.908424 (+ 3621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:30:24.911033 (+ 2609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:30:24.911043 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:30:24.911063 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:30:24.911472 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:30:24.911905 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:30:24.911911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:30:24.911915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:30:24.912116 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:30:24.912733 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:30:24.912740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:30:24.913096 (+ 356us) client_negotiation.cc:769] Sending connection context
0810 22:30:24.914356 (+ 1260us) client_negotiation.cc:240] Negotiation successful
0810 22:30:24.914769 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1758,"thread_start_us":164,"threads_started":1}
I20260810 22:30:24.917515 14748 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33015
I20260810 22:30:24.917791 14748 heartbeater.cc:464] Registering TS with master...
I20260810 22:30:24.918331 14748 heartbeater.cc:511] Master 127.1.221.190:33015 requested a full tablet report, sending...
I20260810 22:30:24.919992 14503 ts_manager.cc:194] Registered new tserver with Master: e1d23dc48365407383453b5dc8eb0635 (127.1.221.129:35549)
I20260810 22:30:24.921280 14503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:50629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:30:24.924180 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:46593/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33015
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:30:24 dist-test-slave-njxd krb5kdc[13971](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786401024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 22:30:25.054803 14754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:30:25.055104 14754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:30:25.055224 14754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:30:25.060685 14754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:30:25.060786 14754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:30:25.060878 14754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:30:25.067179 14754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.221.148:46593/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:30:25.068485 14754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:30:25.069669 14754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:30:25.084399 14761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:30:25.084909 14760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:30:25.085064 14763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:30:25.085475 14754 server_base.cc:1034] running on GCE node
I20260810 22:30:25.086002 14754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:30:25.086838 14754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:30:25.090170 14754 hybrid_clock.cc:648] HybridClock initialized: now 1786401025090122 us; error 52 us; skew 500 ppm
Aug 10 22:30:25 dist-test-slave-njxd krb5kdc[13971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:30:25.093451 14754 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:30:25.094689 14754 webserver.cc:460] Webserver started at http://127.1.221.130:46389/ using document root <none> and password file <none>
I20260810 22:30:25.095353 14754 fs_manager.cc:359] Metadata directory not provided
I20260810 22:30:25.095453 14754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:30:25.095714 14754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:30:25 dist-test-slave-njxd krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:30:25.119500 14754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "00560e01ecf545779fc28d4527a2100b"
format_stamp: "Formatted at 2026-08-10 22:30:25 on dist-test-slave-njxd"
server_key: "627d340f15c9dd21f6eb53a67d2ad314"
server_key_iv: "81a36377648a166993b7761834a54fb1"
server_key_version: "kuduclusterkey@0"
I20260810 22:30:25.120294 14754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "00560e01ecf545779fc28d4527a2100b"
format_stamp: "Formatted at 2026-08-10 22:30:25 on dist-test-slave-njxd"
server_key: "627d340f15c9dd21f6eb53a67d2ad314"
server_key_iv: "81a36377648a166993b7761834a54fb1"
server_key_version: "kuduclusterkey@0"
I20260810 22:30:25.126406 14754 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260810 22:30:25.150876 14770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:30:25.152707 14754 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260810 22:30:25.152859 14754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "00560e01ecf545779fc28d4527a2100b"
format_stamp: "Formatted at 2026-08-10 22:30:25 on dist-test-slave-njxd"
server_key: "627d340f15c9dd21f6eb53a67d2ad314"
server_key_iv: "81a36377648a166993b7761834a54fb1"
server_key_version: "kuduclusterkey@0"
I20260810 22:30:25.152999 14754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:30:25.180892 14754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:30:25.184751 14754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:30:25.185005 14754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:30:25.185886 14754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:30:25.187170 14754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:30:25.187278 14754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:30:25.187382 14754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:30:25.187448 14754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:30:25.200562 14754 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:37943
I20260810 22:30:25.201618 14896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:37943 every 8 connection(s)
I20260810 22:30:25.201946 14754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:30:25.207602 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 14754
I20260810 22:30:25.207757 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:30:25.208088 1910 external_mini_cluster.cc:1428] Setting key 48571e253fe3f70bdcc1798c5700f93e
Aug 10 22:30:25 dist-test-slave-njxd krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:30:25.224460 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.206396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:59439 (local address 127.1.221.190:33015)
0810 22:30:25.206528 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 22:30:25.206533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:30:25.207258 (+ 725us) server_negotiation.cc:407] Connection header received
0810 22:30:25.208569 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:30:25.208573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:30:25.208635 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:30:25.208695 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:30:25.210848 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.211681 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.213607 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.213989 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.214084 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.219019 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:30:25.219024 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:30:25.219027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:30:25.219071 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:30:25.220974 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:30:25.221833 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:30:25.221837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:30:25.221839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:30:25.221935 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:30:25.222977 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:30:25.222981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:30:25.222983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:30:25.223253 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:30:25.223353 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 22:30:25.224103 (+ 750us) server_negotiation.cc:299] Negotiation successful
0810 22:30:25.224262 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:30:25.226480 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.205231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33015 (local address 127.1.221.130:59439)
0810 22:30:25.207087 (+ 1856us) negotiation.cc:107] Waiting for socket to connect
0810 22:30:25.207127 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:30:25.208296 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:30:25.209130 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:30:25.209139 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:30:25.209526 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:30:25.210667 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.210690 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.211856 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.211862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.213425 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.213439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.214309 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.214316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.214571 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.216860 (+ 2289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:30:25.216881 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:30:25.218777 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:30:25.221206 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:30:25.221213 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:30:25.221233 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:30:25.221697 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:30:25.222650 (+ 953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:30:25.222654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:30:25.222656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:30:25.222851 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:30:25.223431 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:30:25.223438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:30:25.223787 (+ 349us) client_negotiation.cc:769] Sending connection context
0810 22:30:25.224957 (+ 1170us) client_negotiation.cc:240] Negotiation successful
0810 22:30:25.225339 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1669,"thread_start_us":146,"threads_started":1}
I20260810 22:30:25.228223 14897 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33015
I20260810 22:30:25.228487 14897 heartbeater.cc:464] Registering TS with master...
I20260810 22:30:25.229148 14897 heartbeater.cc:511] Master 127.1.221.190:33015 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260810 22:30:25.230481 14503 ts_manager.cc:194] Registered new tserver with Master: 00560e01ecf545779fc28d4527a2100b (127.1.221.130:37943)
I20260810 22:30:25.231312 14503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:59439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:30:25.259763 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:46593/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33015
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:44277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:30:25.434010 14903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:30:25.434423 14903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:30:25.434532 14903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:30:25.440828 14903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:30:25.440982 14903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:30:25.441123 14903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:30:25.448726 14903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:44277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.221.148:46593/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:33015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:30:25.450450 14903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:30:25.452014 14903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:30:25.465054 14910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:30:25.465755 14909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:30:25.466050 14912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:30:25.466341 14903 server_base.cc:1034] running on GCE node
I20260810 22:30:25.467036 14903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:30:25.467808 14903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:30:25.468964 14903 hybrid_clock.cc:648] HybridClock initialized: now 1786401025468897 us; error 85 us; skew 500 ppm
Aug 10 22:30:25 dist-test-slave-njxd krb5kdc[13971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:30:25.472674 14903 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:30:25.473801 14903 webserver.cc:460] Webserver started at http://127.1.221.131:35963/ using document root <none> and password file <none>
I20260810 22:30:25.474560 14903 fs_manager.cc:359] Metadata directory not provided
I20260810 22:30:25.474711 14903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:30:25.475003 14903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:30:25 dist-test-slave-njxd krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:30:25.497776 14903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "cc54ced4394a49639492c3fa1b532f1f"
format_stamp: "Formatted at 2026-08-10 22:30:25 on dist-test-slave-njxd"
server_key: "f4849c5d7d7ff73f728db13acab998d4"
server_key_iv: "4d96c337feb230ab672c03517d38f41d"
server_key_version: "kuduclusterkey@0"
I20260810 22:30:25.500396 14903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "cc54ced4394a49639492c3fa1b532f1f"
format_stamp: "Formatted at 2026-08-10 22:30:25 on dist-test-slave-njxd"
server_key: "f4849c5d7d7ff73f728db13acab998d4"
server_key_iv: "4d96c337feb230ab672c03517d38f41d"
server_key_version: "kuduclusterkey@0"
I20260810 22:30:25.508165 14903 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.002s
Aug 10 22:30:25 dist-test-slave-njxd krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:30:25.530745 14920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:30:25.532438 14903 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:30:25.532577 14903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "cc54ced4394a49639492c3fa1b532f1f"
format_stamp: "Formatted at 2026-08-10 22:30:25 on dist-test-slave-njxd"
server_key: "f4849c5d7d7ff73f728db13acab998d4"
server_key_iv: "4d96c337feb230ab672c03517d38f41d"
server_key_version: "kuduclusterkey@0"
I20260810 22:30:25.532722 14903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:30:25.558350 14903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:30:25.562033 14903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:30:25.562229 14903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:30:25.562911 14903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:30:25.563944 14903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:30:25.564009 14903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:30:25.564065 14903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:30:25.564095 14903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:30:25.574522 14903 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:44911
I20260810 22:30:25.575721 14903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:30:25.580252 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 14903
I20260810 22:30:25.580407 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:30:25.580729 1910 external_mini_cluster.cc:1428] Setting key deaeb6775755dd1558a79b10e093b2fe
I20260810 22:30:25.586627 15049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:44911 every 8 connection(s)
Aug 10 22:30:25 dist-test-slave-njxd krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401025, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:30:25.622520 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.597213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:53365 (local address 127.1.221.190:33015)
0810 22:30:25.597331 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 22:30:25.597336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:30:25.598159 (+ 823us) server_negotiation.cc:407] Connection header received
0810 22:30:25.599383 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:30:25.599386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:30:25.599441 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:30:25.599493 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:30:25.601203 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.602086 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.613742 (+ 11656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.614143 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.614297 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.617928 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:30:25.617932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:30:25.617935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:30:25.617967 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:30:25.619795 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:30:25.620533 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:30:25.620537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:30:25.620539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:30:25.620664 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:30:25.621104 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:30:25.621107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:30:25.621109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:30:25.621338 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:30:25.621420 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 22:30:25.622210 (+ 790us) server_negotiation.cc:299] Negotiation successful
0810 22:30:25.622349 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 22:30:25.624655 15052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.596087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:33015 (local address 127.1.221.131:53365)
0810 22:30:25.597973 (+ 1886us) negotiation.cc:107] Waiting for socket to connect
0810 22:30:25.598031 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 22:30:25.599133 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:30:25.599935 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:30:25.599944 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:30:25.600328 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:30:25.601020 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.601041 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.602253 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.602256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.603704 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.603714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.614602 (+ 10888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.614609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.614899 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.615753 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:30:25.615778 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:30:25.617689 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:30:25.619956 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:30:25.619963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:30:25.619986 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:30:25.620414 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:30:25.620780 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:30:25.620783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:30:25.620785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:30:25.620990 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:30:25.621491 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:30:25.621500 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:30:25.621905 (+ 405us) client_negotiation.cc:769] Sending connection context
0810 22:30:25.623021 (+ 1116us) client_negotiation.cc:240] Negotiation successful
0810 22:30:25.623409 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1665,"thread_start_us":162,"threads_started":1}
I20260810 22:30:25.626369 15050 heartbeater.cc:347] Connected to a master server at 127.1.221.190:33015
I20260810 22:30:25.626636 15050 heartbeater.cc:464] Registering TS with master...
I20260810 22:30:25.627234 15050 heartbeater.cc:511] Master 127.1.221.190:33015 requested a full tablet report, sending...
I20260810 22:30:25.628435 14503 ts_manager.cc:194] Registered new tserver with Master: cc54ced4394a49639492c3fa1b532f1f (127.1.221.131:44911)
I20260810 22:30:25.629205 14503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:53365
I20260810 22:30:25.632149 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:30:25.656354 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.642065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.1.221.190:33015)
0810 22:30:25.642224 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 22:30:25.642229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:30:25.643016 (+ 787us) server_negotiation.cc:407] Connection header received
0810 22:30:25.643239 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:30:25.643243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:30:25.643293 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:30:25.643346 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:30:25.644341 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.645185 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.649763 (+ 4578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.650158 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.650255 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.652376 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:30:25.652380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:30:25.652383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:30:25.652419 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:30:25.654318 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:30:25.654906 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:30:25.654910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:30:25.654912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:30:25.655002 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:30:25.655313 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:30:25.655316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:30:25.655318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:30:25.655551 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:30:25.655666 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 22:30:25.656045 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 22:30:25.656181 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 22:30:25.660655 14503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:30:25.665292 14503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:30:25.689383 15063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.680683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:35549 (local address 127.0.0.1:49462)
0810 22:30:25.682630 (+ 1947us) negotiation.cc:107] Waiting for socket to connect
0810 22:30:25.682641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:30:25.682704 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:30:25.684783 (+ 2079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:30:25.684786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:30:25.684802 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:30:25.684892 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.684897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.686321 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.686324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.687550 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.687562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.688834 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.688837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.688966 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.689003 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 22:30:25.689114 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 22:30:25.689233 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1893,"thread_start_us":76,"threads_started":1}
I20260810 22:30:25.689864 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.680863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.1.221.129:35549)
0810 22:30:25.684163 (+ 3300us) server_negotiation.cc:206] Beginning negotiation
0810 22:30:25.684170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:30:25.684197 (+ 27us) server_negotiation.cc:407] Connection header received
0810 22:30:25.684273 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:30:25.684279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:30:25.684466 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:30:25.684634 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:30:25.685019 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.686181 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.687711 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.688500 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.688615 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.688768 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 22:30:25.689619 (+ 851us) server_negotiation.cc:299] Negotiation successful
0810 22:30:25.689731 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3186,"thread_start_us":131,"threads_started":1}
I20260810 22:30:25.690630 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.677657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:37943 (local address 127.0.0.1:53004)
0810 22:30:25.682410 (+ 4753us) negotiation.cc:107] Waiting for socket to connect
0810 22:30:25.682421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:30:25.682473 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:30:25.683589 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:30:25.683596 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:30:25.683618 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:30:25.683755 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.683761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.686033 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.686037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.688179 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.688190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.690120 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.690126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.690250 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.690284 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 22:30:25.690370 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 22:30:25.690456 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3399,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260810 22:30:25.692256 15064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.678674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53004 (local address 127.1.221.130:37943)
0810 22:30:25.682930 (+ 4256us) server_negotiation.cc:206] Beginning negotiation
0810 22:30:25.682947 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:30:25.682971 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:30:25.683046 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:30:25.683051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:30:25.683274 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:30:25.683422 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:30:25.683881 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.685897 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.688343 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.689253 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.689374 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.690220 (+ 846us) server_negotiation.cc:1035] Waiting for connection context
0810 22:30:25.691933 (+ 1713us) server_negotiation.cc:299] Negotiation successful
0810 22:30:25.692064 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4148,"thread_start_us":69,"threads_started":1}
I20260810 22:30:25.693274 14675 tablet_service.cc:1467] Processing CreateTablet for tablet f40328b4f77a4f52aa9e9a231329b233 (DEFAULT_TABLE table=test-table [id=4e7df49e2ce64890a36b30dfd5d2b20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:30:25.694648 14675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40328b4f77a4f52aa9e9a231329b233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:30:25.695327 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.680339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:44911 (local address 127.0.0.1:53874)
0810 22:30:25.682105 (+ 1766us) negotiation.cc:107] Waiting for socket to connect
0810 22:30:25.682160 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 22:30:25.682289 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:30:25.690965 (+ 8676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:30:25.690968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:30:25.690987 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:30:25.691070 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.691076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.692564 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.692568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.693679 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.693691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.694879 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.694883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.694971 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.695054 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 22:30:25.695137 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 22:30:25.695203 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1615,"thread_start_us":93,"threads_started":1}
I20260810 22:30:25.695210 14824 tablet_service.cc:1467] Processing CreateTablet for tablet f40328b4f77a4f52aa9e9a231329b233 (DEFAULT_TABLE table=test-table [id=4e7df49e2ce64890a36b30dfd5d2b20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:30:25.696290 14824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40328b4f77a4f52aa9e9a231329b233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:30:25.696732 15066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.680531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.1.221.131:44911)
0810 22:30:25.690470 (+ 9939us) server_negotiation.cc:206] Beginning negotiation
0810 22:30:25.690482 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:30:25.690505 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:30:25.690576 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:30:25.690581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:30:25.690793 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:30:25.691233 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:30:25.691377 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.692424 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.693861 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.694751 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.694860 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.694992 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 22:30:25.696486 (+ 1494us) server_negotiation.cc:299] Negotiation successful
0810 22:30:25.696587 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9823,"thread_start_us":123,"threads_started":1}
I20260810 22:30:25.699420 14970 tablet_service.cc:1467] Processing CreateTablet for tablet f40328b4f77a4f52aa9e9a231329b233 (DEFAULT_TABLE table=test-table [id=4e7df49e2ce64890a36b30dfd5d2b20e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:30:25.700460 14970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40328b4f77a4f52aa9e9a231329b233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:30:25.705288 15071 tablet_bootstrap.cc:492] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b: Bootstrap starting.
I20260810 22:30:25.707829 15069 tablet_bootstrap.cc:492] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635: Bootstrap starting.
I20260810 22:30:25.710319 15072 tablet_bootstrap.cc:492] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f: Bootstrap starting.
I20260810 22:30:25.710361 15069 tablet_bootstrap.cc:654] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635: Neither blocks nor log segments found. Creating new log.
I20260810 22:30:25.711370 15069 log.cc:824] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635: Log is configured to *not* fsync() on all Append() calls
I20260810 22:30:25.713119 15072 tablet_bootstrap.cc:654] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f: Neither blocks nor log segments found. Creating new log.
I20260810 22:30:25.714053 15072 log.cc:824] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:30:25.716370 15072 tablet_bootstrap.cc:492] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f: No bootstrap required, opened a new log
I20260810 22:30:25.716586 15072 ts_tablet_manager.cc:1397] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:30:25.720871 15072 raft_consensus.cc:348] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00560e01ecf545779fc28d4527a2100b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37943 } } peers { permanent_uuid: "e1d23dc48365407383453b5dc8eb0635" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35549 } } peers { permanent_uuid: "cc54ced4394a49639492c3fa1b532f1f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44911 } }
I20260810 22:30:25.721165 15072 raft_consensus.cc:374] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:30:25.721218 15072 raft_consensus.cc:729] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc54ced4394a49639492c3fa1b532f1f, State: Initialized, Role: FOLLOWER
I20260810 22:30:25.722482 15071 tablet_bootstrap.cc:654] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b: Neither blocks nor log segments found. Creating new log.
I20260810 22:30:25.723219 15072 consensus_queue.cc:260] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00560e01ecf545779fc28d4527a2100b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37943 } } peers { permanent_uuid: "e1d23dc48365407383453b5dc8eb0635" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35549 } } peers { permanent_uuid: "cc54ced4394a49639492c3fa1b532f1f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44911 } }
I20260810 22:30:25.724073 15071 log.cc:824] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b: Log is configured to *not* fsync() on all Append() calls
I20260810 22:30:25.724568 15072 ts_tablet_manager.cc:1428] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 22:30:25.724982 15050 heartbeater.cc:503] Master 127.1.221.190:33015 was elected leader, sending a full tablet report...
I20260810 22:30:25.735455 15071 tablet_bootstrap.cc:492] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b: No bootstrap required, opened a new log
I20260810 22:30:25.735621 15069 tablet_bootstrap.cc:492] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635: No bootstrap required, opened a new log
I20260810 22:30:25.735706 15071 ts_tablet_manager.cc:1397] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.000s
I20260810 22:30:25.735782 15069 ts_tablet_manager.cc:1397] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20260810 22:30:25.740228 15071 raft_consensus.cc:348] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00560e01ecf545779fc28d4527a2100b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37943 } } peers { permanent_uuid: "e1d23dc48365407383453b5dc8eb0635" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35549 } } peers { permanent_uuid: "cc54ced4394a49639492c3fa1b532f1f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44911 } }
I20260810 22:30:25.740638 15071 raft_consensus.cc:374] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:30:25.740762 15071 raft_consensus.cc:729] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00560e01ecf545779fc28d4527a2100b, State: Initialized, Role: FOLLOWER
I20260810 22:30:25.741381 15071 consensus_queue.cc:260] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00560e01ecf545779fc28d4527a2100b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37943 } } peers { permanent_uuid: "e1d23dc48365407383453b5dc8eb0635" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35549 } } peers { permanent_uuid: "cc54ced4394a49639492c3fa1b532f1f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44911 } }
I20260810 22:30:25.742667 15071 ts_tablet_manager.cc:1428] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 22:30:25.743077 14897 heartbeater.cc:503] Master 127.1.221.190:33015 was elected leader, sending a full tablet report...
I20260810 22:30:25.744058 15069 raft_consensus.cc:348] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00560e01ecf545779fc28d4527a2100b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37943 } } peers { permanent_uuid: "e1d23dc48365407383453b5dc8eb0635" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35549 } } peers { permanent_uuid: "cc54ced4394a49639492c3fa1b532f1f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44911 } }
I20260810 22:30:25.744338 15069 raft_consensus.cc:374] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:30:25.744396 15069 raft_consensus.cc:729] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d23dc48365407383453b5dc8eb0635, State: Initialized, Role: FOLLOWER
I20260810 22:30:25.744916 15069 consensus_queue.cc:260] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00560e01ecf545779fc28d4527a2100b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37943 } } peers { permanent_uuid: "e1d23dc48365407383453b5dc8eb0635" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35549 } } peers { permanent_uuid: "cc54ced4394a49639492c3fa1b532f1f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44911 } }
I20260810 22:30:25.746502 15069 ts_tablet_manager.cc:1428] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260810 22:30:25.747056 14748 heartbeater.cc:503] Master 127.1.221.190:33015 was elected leader, sending a full tablet report...
W20260810 22:30:25.848003 15051 tablet.cc:2369] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 22:30:25 dist-test-slave-njxd krb5kdc[13971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
W20260810 22:30:25.890872 14749 tablet.cc:2369] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:30:25.937391 15078 raft_consensus.cc:482] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:30:25.937695 15078 raft_consensus.cc:504] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00560e01ecf545779fc28d4527a2100b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37943 } } peers { permanent_uuid: "e1d23dc48365407383453b5dc8eb0635" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35549 } } peers { permanent_uuid: "cc54ced4394a49639492c3fa1b532f1f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44911 } }
I20260810 22:30:25.939167 15078 leader_election.cc:288] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00560e01ecf545779fc28d4527a2100b (127.1.221.130:37943), cc54ced4394a49639492c3fa1b532f1f (127.1.221.131:44911)
I20260810 22:30:25.939316 15075 raft_consensus.cc:482] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:30:25.939503 15075 raft_consensus.cc:504] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00560e01ecf545779fc28d4527a2100b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37943 } } peers { permanent_uuid: "e1d23dc48365407383453b5dc8eb0635" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35549 } } peers { permanent_uuid: "cc54ced4394a49639492c3fa1b532f1f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44911 } }
I20260810 22:30:25.940565 15075 leader_election.cc:288] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00560e01ecf545779fc28d4527a2100b (127.1.221.130:37943), e1d23dc48365407383453b5dc8eb0635 (127.1.221.129:35549)
I20260810 22:30:25.955328 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.941727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:44755 (local address 127.1.221.129:35549)
0810 22:30:25.941837 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 22:30:25.941841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:30:25.950982 (+ 9141us) server_negotiation.cc:407] Connection header received
0810 22:30:25.951035 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:30:25.951038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:30:25.951078 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:30:25.951156 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:30:25.951580 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.952332 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.953968 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.954531 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.954671 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.954727 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:30:25.955078 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 22:30:25.955161 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":34}
I20260810 22:30:25.955844 15064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.940440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:54415 (local address 127.1.221.130:37943)
0810 22:30:25.940788 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0810 22:30:25.940793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:30:25.949912 (+ 9119us) server_negotiation.cc:407] Connection header received
0810 22:30:25.951421 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:30:25.951425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:30:25.951478 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:30:25.951531 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:30:25.952048 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.952784 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.954501 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.955138 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.955223 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.955269 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 22:30:25.955629 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 22:30:25.955712 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260810 22:30:25.956233 15083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.941359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:35549 (local address 127.1.221.131:44755)
0810 22:30:25.950741 (+ 9382us) negotiation.cc:107] Waiting for socket to connect
0810 22:30:25.950758 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:30:25.950878 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:30:25.951282 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:30:25.951286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:30:25.951305 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:30:25.951452 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.951459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.952466 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.952469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.953827 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.953839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.954837 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.954840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.954942 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.954965 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:30:25.955813 (+ 848us) client_negotiation.cc:240] Negotiation successful
0810 22:30:25.956050 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9300,"thread_start_us":97,"threads_started":1}
I20260810 22:30:25.956626 15082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.939550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:37943 (local address 127.1.221.129:54415)
0810 22:30:25.949775 (+ 10225us) negotiation.cc:107] Waiting for socket to connect
0810 22:30:25.949798 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 22:30:25.950022 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:30:25.951734 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:30:25.951736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:30:25.951753 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:30:25.951865 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.951871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.952990 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.952993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.954366 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.954376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.955377 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.955380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.955471 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.955497 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:30:25.956384 (+ 887us) client_negotiation.cc:240] Negotiation successful
0810 22:30:25.956461 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10125,"mutex_wait_us":25,"thread_start_us":130,"threads_started":1}
I20260810 22:30:25.957127 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40328b4f77a4f52aa9e9a231329b233" candidate_uuid: "cc54ced4394a49639492c3fa1b532f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d23dc48365407383453b5dc8eb0635" is_pre_election: true
I20260810 22:30:25.957414 14692 raft_consensus.cc:2454] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc54ced4394a49639492c3fa1b532f1f in term 0.
I20260810 22:30:25.957531 14847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40328b4f77a4f52aa9e9a231329b233" candidate_uuid: "e1d23dc48365407383453b5dc8eb0635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00560e01ecf545779fc28d4527a2100b" is_pre_election: true
I20260810 22:30:25.957880 14847 raft_consensus.cc:2454] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1d23dc48365407383453b5dc8eb0635 in term 0.
I20260810 22:30:25.958303 14921 leader_election.cc:302] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc54ced4394a49639492c3fa1b532f1f, e1d23dc48365407383453b5dc8eb0635; no voters:
I20260810 22:30:25.958341 14621 leader_election.cc:302] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00560e01ecf545779fc28d4527a2100b, e1d23dc48365407383453b5dc8eb0635; no voters:
I20260810 22:30:25.958654 15075 raft_consensus.cc:2781] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:30:25.958654 15078 raft_consensus.cc:2781] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:30:25.958734 15075 raft_consensus.cc:482] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:30:25.958734 15078 raft_consensus.cc:482] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:30:25.958783 15078 raft_consensus.cc:3037] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:30:25.958782 15075 raft_consensus.cc:3037] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:30:25.960402 15078 raft_consensus.cc:504] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00560e01ecf545779fc28d4527a2100b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37943 } } peers { permanent_uuid: "e1d23dc48365407383453b5dc8eb0635" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35549 } } peers { permanent_uuid: "cc54ced4394a49639492c3fa1b532f1f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44911 } }
I20260810 22:30:25.960491 15075 raft_consensus.cc:504] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00560e01ecf545779fc28d4527a2100b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37943 } } peers { permanent_uuid: "e1d23dc48365407383453b5dc8eb0635" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35549 } } peers { permanent_uuid: "cc54ced4394a49639492c3fa1b532f1f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44911 } }
I20260810 22:30:25.960677 15066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.940568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:54021 (local address 127.1.221.131:44911)
0810 22:30:25.940879 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0810 22:30:25.940883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:30:25.950524 (+ 9641us) server_negotiation.cc:407] Connection header received
0810 22:30:25.950757 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:30:25.950761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:30:25.950816 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:30:25.950870 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:30:25.957149 (+ 6279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.957805 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.959279 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.959874 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.959967 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.960013 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 22:30:25.960480 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 22:30:25.960568 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20260810 22:30:25.960807 15075 leader_election.cc:288] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [CANDIDATE]: Term 1 election: Requested vote from peers 00560e01ecf545779fc28d4527a2100b (127.1.221.130:37943), e1d23dc48365407383453b5dc8eb0635 (127.1.221.129:35549)
I20260810 22:30:25.960814 15078 leader_election.cc:288] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [CANDIDATE]: Term 1 election: Requested vote from peers 00560e01ecf545779fc28d4527a2100b (127.1.221.130:37943), cc54ced4394a49639492c3fa1b532f1f (127.1.221.131:44911)
I20260810 22:30:25.961359 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.939965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:44911 (local address 127.1.221.129:54021)
0810 22:30:25.950416 (+ 10451us) negotiation.cc:107] Waiting for socket to connect
0810 22:30:25.950434 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:30:25.950620 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:30:25.956659 (+ 6039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:30:25.956665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:30:25.956859 (+ 194us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:30:25.956972 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.956981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.957933 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.957937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.959125 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.959138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.960136 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.960142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.960317 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.960343 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:30:25.960861 (+ 518us) client_negotiation.cc:240] Negotiation successful
0810 22:30:25.960949 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10376,"thread_start_us":90,"threads_started":1}
I20260810 22:30:25.962110 14998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40328b4f77a4f52aa9e9a231329b233" candidate_uuid: "e1d23dc48365407383453b5dc8eb0635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc54ced4394a49639492c3fa1b532f1f"
I20260810 22:30:25.962370 14998 raft_consensus.cc:2379] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e1d23dc48365407383453b5dc8eb0635 in current term 1: Already voted for candidate cc54ced4394a49639492c3fa1b532f1f in this term.
I20260810 22:30:25.962412 14999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40328b4f77a4f52aa9e9a231329b233" candidate_uuid: "e1d23dc48365407383453b5dc8eb0635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc54ced4394a49639492c3fa1b532f1f" is_pre_election: true
I20260810 22:30:25.962538 14847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40328b4f77a4f52aa9e9a231329b233" candidate_uuid: "e1d23dc48365407383453b5dc8eb0635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00560e01ecf545779fc28d4527a2100b"
I20260810 22:30:25.962719 14847 raft_consensus.cc:3037] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:30:25.964221 14692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40328b4f77a4f52aa9e9a231329b233" candidate_uuid: "cc54ced4394a49639492c3fa1b532f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d23dc48365407383453b5dc8eb0635"
I20260810 22:30:25.964308 14847 raft_consensus.cc:2454] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1d23dc48365407383453b5dc8eb0635 in term 1.
I20260810 22:30:25.964401 14692 raft_consensus.cc:2379] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cc54ced4394a49639492c3fa1b532f1f in current term 1: Already voted for candidate e1d23dc48365407383453b5dc8eb0635 in this term.
W20260810 22:30:25.964628 14898 tablet.cc:2369] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:30:25.965332 14621 leader_election.cc:302] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00560e01ecf545779fc28d4527a2100b, e1d23dc48365407383453b5dc8eb0635; no voters: cc54ced4394a49639492c3fa1b532f1f
I20260810 22:30:25.966938 15052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.941117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:37943 (local address 127.1.221.131:57285)
0810 22:30:25.941219 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 22:30:25.941235 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:30:25.941343 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:30:25.956256 (+ 14913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:30:25.956260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:30:25.956284 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:30:25.956394 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.956400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.961003 (+ 4603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.961009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.962086 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:30:25.962096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.966520 (+ 4424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.966524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:30:25.966644 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.966670 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:30:25.966742 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 22:30:25.966817 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 22:30:25.967116 15078 raft_consensus.cc:2781] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:30:25.967355 15078 raft_consensus.cc:686] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [term 1 LEADER]: Becoming Leader. State: Replica: e1d23dc48365407383453b5dc8eb0635, State: Running, Role: LEADER
I20260810 22:30:25.967758 15078 consensus_queue.cc:237] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00560e01ecf545779fc28d4527a2100b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37943 } } peers { permanent_uuid: "e1d23dc48365407383453b5dc8eb0635" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35549 } } peers { permanent_uuid: "cc54ced4394a49639492c3fa1b532f1f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44911 } }
I20260810 22:30:25.969987 15064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:25.941799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:57285 (local address 127.1.221.130:37943)
0810 22:30:25.956028 (+ 14229us) server_negotiation.cc:206] Beginning negotiation
0810 22:30:25.956034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:30:25.956046 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:30:25.956084 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:30:25.956087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:30:25.956148 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:30:25.956229 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:30:25.956535 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.957174 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.965740 (+ 8566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:25.966298 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:25.966414 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:25.966459 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 22:30:25.969754 (+ 3295us) server_negotiation.cc:299] Negotiation successful
0810 22:30:25.969846 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14173,"thread_start_us":101,"threads_started":1}
I20260810 22:30:25.970813 14846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40328b4f77a4f52aa9e9a231329b233" candidate_uuid: "cc54ced4394a49639492c3fa1b532f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00560e01ecf545779fc28d4527a2100b"
I20260810 22:30:25.971024 14846 raft_consensus.cc:2379] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cc54ced4394a49639492c3fa1b532f1f in current term 1: Already voted for candidate e1d23dc48365407383453b5dc8eb0635 in this term.
I20260810 22:30:25.972283 14503 catalog_manager.cc:5495] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1d23dc48365407383453b5dc8eb0635 (127.1.221.129). New cstate: current_term: 1 leader_uuid: "e1d23dc48365407383453b5dc8eb0635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00560e01ecf545779fc28d4527a2100b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1d23dc48365407383453b5dc8eb0635" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 35549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc54ced4394a49639492c3fa1b532f1f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44911 } health_report { overall_health: UNKNOWN } } }
I20260810 22:30:25.975291 14921 leader_election.cc:302] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc54ced4394a49639492c3fa1b532f1f; no voters: 00560e01ecf545779fc28d4527a2100b, e1d23dc48365407383453b5dc8eb0635
I20260810 22:30:25.975756 15075 raft_consensus.cc:2726] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 22:30:25.970836 14847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40328b4f77a4f52aa9e9a231329b233" candidate_uuid: "cc54ced4394a49639492c3fa1b532f1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00560e01ecf545779fc28d4527a2100b" is_pre_election: true
I20260810 22:30:25.976864 14847 raft_consensus.cc:2379] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc54ced4394a49639492c3fa1b532f1f in current term 1: Already voted for candidate e1d23dc48365407383453b5dc8eb0635 in this term.
I20260810 22:30:26.065289 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:26.060099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.1.221.129:35549)
0810 22:30:26.060319 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 22:30:26.060324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:30:26.060368 (+ 44us) server_negotiation.cc:407] Connection header received
0810 22:30:26.060583 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:30:26.060586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:30:26.060664 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:30:26.060790 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:30:26.061187 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:26.061976 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:26.063261 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:30:26.063594 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:30:26.063707 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:30:26.064120 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:30:26.064679 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:30:26.064804 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 22:30:26.065058 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 22:30:26.065111 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260810 22:30:26.081840 14999 raft_consensus.cc:1261] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 1 FOLLOWER]: Refusing update from remote peer e1d23dc48365407383453b5dc8eb0635: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:30:26.082459 15078 consensus_queue.cc:1035] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc54ced4394a49639492c3fa1b532f1f" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 44911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:30:26.090082 14847 raft_consensus.cc:1261] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b [term 1 FOLLOWER]: Refusing update from remote peer e1d23dc48365407383453b5dc8eb0635: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:30:26.090632 15078 consensus_queue.cc:1035] T f40328b4f77a4f52aa9e9a231329b233 P e1d23dc48365407383453b5dc8eb0635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00560e01ecf545779fc28d4527a2100b" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:30:26.104928 15089 mvcc.cc:205] Tried to move back new op lower bound from 7317098602807123968 to 7317098602368110592. Current Snapshot: MvccSnapshot[applied={T|T < 7317098602807123968}]
I20260810 22:30:26.109561 15092 mvcc.cc:205] Tried to move back new op lower bound from 7317098602807123968 to 7317098602368110592. Current Snapshot: MvccSnapshot[applied={T|T < 7317098602807123968}]
I20260810 22:30:26.114889 15090 mvcc.cc:205] Tried to move back new op lower bound from 7317098602807123968 to 7317098602368110592. Current Snapshot: MvccSnapshot[applied={T|T < 7317098602807123968}]
I20260810 22:30:26.126757 14510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:30:26.126956 14510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:30:26.129570 14510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4e4481eac1d4c318fa5510b555db9f9: Sending DeleteTablet for 3 replicas of tablet f40328b4f77a4f52aa9e9a231329b233
I20260810 22:30:26.130649 14675 tablet_service.cc:1514] Processing DeleteTablet for tablet f40328b4f77a4f52aa9e9a231329b233 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:30:26 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:49462
I20260810 22:30:26.130641 14824 tablet_service.cc:1514] Processing DeleteTablet for tablet f40328b4f77a4f52aa9e9a231329b233 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:30:26 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:53004
I20260810 22:30:26.131556 14970 tablet_service.cc:1514] Processing DeleteTablet for tablet f40328b4f77a4f52aa9e9a231329b233 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:30:26 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:53874
I20260810 22:30:26.132359 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 14602
I20260810 22:30:26.135179 15101 tablet_replica.cc:330] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f: stopping tablet replica
I20260810 22:30:26.135578 15101 raft_consensus.cc:2229] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:30:26.135955 15101 raft_consensus.cc:2258] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:30:26.140344 15100 tablet_replica.cc:330] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b: stopping tablet replica
I20260810 22:30:26.140676 15100 raft_consensus.cc:2229] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:30:26.141000 15100 raft_consensus.cc:2258] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:30:26.142858 15101 ts_tablet_manager.cc:1905] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:30:26.146682 15101 ts_tablet_manager.cc:1918] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:30:26.146838 15101 log.cc:1196] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/f40328b4f77a4f52aa9e9a231329b233
I20260810 22:30:26.147202 15101 ts_tablet_manager.cc:1939] T f40328b4f77a4f52aa9e9a231329b233 P cc54ced4394a49639492c3fa1b532f1f: Deleting consensus metadata
W20260810 22:30:26.148379 14490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:35549 (error 108)
I20260810 22:30:26.148936 14494 catalog_manager.cc:4841] TS cc54ced4394a49639492c3fa1b532f1f (127.1.221.131:44911): tablet f40328b4f77a4f52aa9e9a231329b233 (table test-table [id=4e7df49e2ce64890a36b30dfd5d2b20e]) successfully deleted
I20260810 22:30:26.150440 15100 ts_tablet_manager.cc:1905] T f40328b4f77a4f52aa9e9a231329b233 P 00560e01ecf545779fc28d4527a2100b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:30:26.150549 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:26.150264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:35549 (local address 127.0.0.1:49468)
0810 22:30:26.150382 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 22:30:26.150457 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:35549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 22:30:26.150542 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 14754
W20260810 22:30:26.152130 14490 catalog_manager.cc:4584] TS e1d23dc48365407383453b5dc8eb0635 (127.1.221.129:35549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f40328b4f77a4f52aa9e9a231329b233: Network error: Client connection negotiation failed: client connection to 127.1.221.129:35549: connect: Connection refused (error 111)
I20260810 22:30:26.161286 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 14903
I20260810 22:30:26.169061 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:30:26.168730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:37943 (local address 127.0.0.1:53014)
0810 22:30:26.168880 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 22:30:26.168962 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.130:37943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 22:30:26.175510 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 14474
I20260810 22:30:26.184242 1910 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 22:30:26.530900 1910 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 22:30:26.531051 1910 mini_ranger.cc:66] Stopping Ranger...
I20260810 22:30:26.886431 1910 mini_ranger.cc:68] Stopped Ranger
2026-08-10 22:30:26.886 UTC [14004] LOG: received smart shutdown request
2026-08-10 22:30:26.890 UTC [14004] LOG: background worker "logical replication launcher" (PID 14013) exited with exit code 1
2026-08-10 22:30:26.891 UTC [14008] LOG: shutting down
2026-08-10 22:30:27.047 UTC [14004] LOG: database system is shut down
2026-08-10T22:30:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48038 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:30:27 dist-test-slave-njxd krb5kdc[15122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:30:27 dist-test-slave-njxd krb5kdc[15122](info): set up 2 sockets
Aug 10 22:30:27 dist-test-slave-njxd krb5kdc[15122](info): commencing operation
krb5kdc: starting...
W20260810 22:30:28.267938 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:30:28 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:30:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:30:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/rangeradmin_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/rangeradmin_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/rangerlookup_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/rangerlookup_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
I20260810 22:30:28.448137 1910 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYvFf7I/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/postgres -l logfile start
2026-08-10 22:30:30.836 UTC [15153] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 22:30:30.836 UTC [15153] LOG: listening on IPv4 address "127.1.221.148", port 55753
2026-08-10 22:30:30.840 UTC [15153] LOG: listening on Unix socket "/tmp/.s.PGSQL.55753"
2026-08-10 22:30:30.877 UTC [15156] LOG: database system was shut down at 2026-08-10 22:30:29 UTC
2026-08-10 22:30:30.881 UTC [15153] LOG: database system is ready to accept connections
I20260810 22:30:31.873999 1910 mini_postgres.cc:96] Postgres bound to 55753
2026-08-10 22:30:31.881 UTC [15164] FATAL: database "slave" does not exist
127.1.221.148:55753 - accepting connections
I20260810 22:30:31.881896 1910 mini_ranger.cc:161] Starting Ranger...
I20260810 22:30:31.897557 1910 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 22:30:32.364559 1910 mini_ranger.cc:87] Created ranger Postgres database
I20260810 22:30:32.364653 1910 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-admin
2026-08-10 22:30:32,656 [I] DB FLAVOR :POSTGRES
2026-08-10 22:30:32,657 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 22:30:32,657 [I] Checking connection..
2026-08-10 22:30:32,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 22:30:32,911 [I] Checking connection passed.
2026-08-10 22:30:32,911 [I] --------- Verifying version history table ---------
2026-08-10 22:30:32,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:30:33,164 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 22:30:33,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:30:33,419 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 22:30:33,420 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 22:30:33,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 22:30:33.670 UTC [15234] WARNING: there is no transaction in progress
2026-08-10 22:30:33,679 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 22:30:33,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:30:33,932 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 22:30:33,933 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 22:30:33,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 22:30:34,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 22:30:34,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-njxd.c.gcp-upstream.internal','N') ;"
2026-08-10 22:30:34,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 22:30:34,949 [I] Table x_portal_user does not exist in database ranger
2026-08-10 22:30:34,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 22:30:35,202 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 22:30:35,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 22:30:35,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 22:30:35,717 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 22:30:35,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 22:30:36.677 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.693 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.709 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.724 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.739 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.757 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.815 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.821 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.836 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.846 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.856 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.869 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.878 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.885 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.892 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:36.898 UTC [15355] WARNING: there is no transaction in progress
2026-08-10 22:30:37,496 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 22:30:37,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 22:30:37,757 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 22:30:37,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 22:30:37,997 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 22:30:37,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 22:30:38,253 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 22:30:38,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 22:30:38,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 22:30:38,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-njxd.c.gcp-upstream.internal';"
2026-08-10 22:30:38,994 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 22:30:38,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 22:30:39,240 [I] DB_PATCHES have already been applied
I20260810 22:30:39.246050 1910 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYvFf7I/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 22:30:39.246114 1910 mini_ranger.cc:193] Using host: 127.1.221.148
I20260810 22:30:39.250782 1910 mini_ranger.cc:227] Ranger admin URL: http://127.1.221.148:42595
Aug 10, 2026 10:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 10:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 10:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 10:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 10:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 10:30:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 10:30:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 22:30:40.462591 1910 mini_ranger.cc:160] Time spent starting Ranger: real 8.581s user 0.001s sys 0.010s
Aug 10, 2026 10:31:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 10:31:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 10:31:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 10:31:01 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 10:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 10:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 10:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 10:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 10:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 10:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 10:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 10:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 10:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 10:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 10:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 10:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 10:31:01 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 10:31:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 22:31:02.798588 1910 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.221.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.221.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/rangerkms_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.221.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/rangerkms_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.221.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.221.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.221.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/HTTP_127.1.221.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 22:31:02 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401062, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 22:31:02.896368 1910 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 22:31:02.907476 1910 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 22:31:03.748178 1910 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 22:31:03.748265 1910 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-kms
I20260810 22:31:03.748281 1910 mini_ranger_kms.cc:227] Using postgres at 127.1.221.148:55753
2026-08-10 22:31:03,927 [I] DB FLAVOR :POSTGRES
2026-08-10 22:31:03,928 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 22:31:03,928 [I] Checking connection
2026-08-10 22:31:03,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 22:31:04,193 [I] connection success
2026-08-10 22:31:04,193 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 22:31:04,193 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 22:31:04,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 22:31:04,442 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 22:31:04,442 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 22:31:04,442 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 22:31:04,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.221.148:55753/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 22:31:04,717 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 22:31:05.240868 1910 mini_ranger_kms.cc:310] Created kms service
I20260810 22:31:05.410843 1910 mini_ranger_kms.cc:326] Created kudu user
I20260810 22:31:05.456856 1910 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 22:31:05 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401062, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:31:05.880125 1910 mini_ranger_kms.cc:384] Added ranger policy
I20260810 22:31:05.880299 1910 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYvFf7I/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYvFf7I/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 22:31:05.880323 1910 mini_ranger_kms.cc:240] Using host: 127.1.221.148
I20260810 22:31:05.885444 1910 mini_ranger_kms.cc:276] Ranger KMS PID: 15567
I20260810 22:31:05.885530 1910 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.221.148:41267
Aug 10, 2026 10:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 10:31:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 10:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 10:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 10:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 10:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 22:31:08.238173 1910 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.342s user 0.006s sys 0.159s
I20260810 22:31:08.238304 1910 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 22:31:08.238389 1910 mini_ranger_kms.cc:401] 127.1.221.148:41267/kms/v1/keys
Aug 10 22:31:08 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786401068, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.221.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 22:31:08 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401068, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.221.148@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
Aug 10 22:31:09 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401068, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.221.148@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
Aug 10 22:31:11 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:31:11.377346 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40093
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40093
--ranger_config_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:41267/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:11.494194 15618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:11.494508 15618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:11.494653 15618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:11.498845 15618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:31:11.498935 15618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:11.498973 15618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:11.499030 15618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:31:11.499089 15618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:31:11.505275 15618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.221.148:41267/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40093
--ranger_config_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:11.506500 15618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:11.507612 15618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:31:11.516988 15618 server_base.cc:1034] running on GCE node
W20260810 22:31:11.517011 15626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:11.517022 15625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:11.517011 15628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:11.517937 15618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:11.519124 15618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:11.520300 15618 hybrid_clock.cc:648] HybridClock initialized: now 1786401071520274 us; error 42 us; skew 500 ppm
Aug 10 22:31:11 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401071, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:11.523250 15618 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:31:11.524257 15618 webserver.cc:460] Webserver started at http://127.1.221.190:38783/ using document root <none> and password file <none>
I20260810 22:31:11.524837 15618 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:11.524895 15618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:11.525134 15618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:31:11 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401071, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:31:11.584650 15618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "eaf8b44dd62d4d31bcb5380df19e5f20"
format_stamp: "Formatted at 2026-08-10 22:31:11 on dist-test-slave-njxd"
server_key: "04e749086d703eb378caeaff1e027b2c"
server_key_iv: "1ebdedbdcaceb70bced7821965bd6306"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:11.585430 15618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "eaf8b44dd62d4d31bcb5380df19e5f20"
format_stamp: "Formatted at 2026-08-10 22:31:11 on dist-test-slave-njxd"
server_key: "04e749086d703eb378caeaff1e027b2c"
server_key_iv: "1ebdedbdcaceb70bced7821965bd6306"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:11.590202 15618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:31:11.610504 15636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:11.612077 15618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:31:11.612257 15618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "eaf8b44dd62d4d31bcb5380df19e5f20"
format_stamp: "Formatted at 2026-08-10 22:31:11 on dist-test-slave-njxd"
server_key: "04e749086d703eb378caeaff1e027b2c"
server_key_iv: "1ebdedbdcaceb70bced7821965bd6306"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:11.612411 15618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:11.643707 15618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:11.647046 15618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:11.647284 15618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:11.655009 15618 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:40093
I20260810 22:31:11.655015 15696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:40093 every 8 connection(s)
I20260810 22:31:11.656131 15618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:31:11.656589 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 15618
I20260810 22:31:11.656739 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:31:11.657039 1910 external_mini_cluster.cc:1428] Setting key 2ecd6322475a149952e0c0d534285106
I20260810 22:31:11.659500 15698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:11.666595 15698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20: Bootstrap starting.
Aug 10 22:31:11 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:11.668968 15698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:11.670181 15698 log.cc:824] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:11.672226 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:11.658618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.1.221.190:40093)
0810 22:31:11.659061 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:11.659083 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:11.659121 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:31:11.659854 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:11.659874 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:11.660216 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:11.660512 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:11.661447 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:11.662601 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:11.663826 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:11.664304 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:11.664420 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:11.667438 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:11.667443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:11.667457 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:11.667487 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:11.669275 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:11.669809 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:11.669814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:11.669819 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:11.669924 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:11.670217 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:11.670222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:11.670224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:11.670676 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:11.670883 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:11.671197 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 22:31:11.671496 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":155,"threads_started":1}
I20260810 22:31:11.672359 15698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20: No bootstrap required, opened a new log
I20260810 22:31:11.675606 15698 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40093 } }
I20260810 22:31:11.675936 15698 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:11.676019 15698 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf8b44dd62d4d31bcb5380df19e5f20, State: Initialized, Role: FOLLOWER
I20260810 22:31:11.676554 15698 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40093 } }
I20260810 22:31:11.676705 15698 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:31:11.676755 15698 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:31:11.676829 15698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:11.677831 15698 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40093 } }
I20260810 22:31:11.678141 15698 leader_election.cc:302] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaf8b44dd62d4d31bcb5380df19e5f20; no voters:
I20260810 22:31:11.678421 15698 leader_election.cc:288] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:31:11.678573 15703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:31:11.678830 15703 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 1 LEADER]: Becoming Leader. State: Replica: eaf8b44dd62d4d31bcb5380df19e5f20, State: Running, Role: LEADER
I20260810 22:31:11.679248 15703 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40093 } }
I20260810 22:31:11.679808 15698 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:31:11.681118 15704 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40093 } } }
I20260810 22:31:11.681250 15704 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:11.681102 15705 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaf8b44dd62d4d31bcb5380df19e5f20. Latest consensus state: current_term: 1 leader_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40093 } } }
I20260810 22:31:11.681538 15705 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:11.681777 15713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:31:11.682209 15698 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 22:31:11.687779 15713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:31:11.696166 15713 catalog_manager.cc:1347] Generated new cluster ID: aceab3d8bbba42c694727ce8cee93af6
I20260810 22:31:11.696249 15713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:31:11.711428 15713 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:31:11.712523 15713 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:31:11.720028 15713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20: Generated new TSK 0
I20260810 22:31:11.720949 15713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:31:12.410159 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:41267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40093
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:12.542704 15747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:12.543004 15747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:12.543078 15747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:12.547369 15747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:12.547541 15747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:12.547678 15747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:31:12.554659 15747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.221.148:41267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:12.556051 15747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:12.557379 15747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:12.569146 15757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:12.569146 15759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:12.569164 15756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:12.569459 15747 server_base.cc:1034] running on GCE node
I20260810 22:31:12.570014 15747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:12.570685 15747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:12.578070 15747 hybrid_clock.cc:648] HybridClock initialized: now 1786401072578028 us; error 55 us; skew 500 ppm
Aug 10 22:31:12 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:12.581310 15747 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:31:12.582361 15747 webserver.cc:460] Webserver started at http://127.1.221.129:39251/ using document root <none> and password file <none>
I20260810 22:31:12.582949 15747 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:12.583010 15747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:12.583227 15747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:31:12 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:31:12.604343 15747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "5401c8c57ca448799399229bb8d1ada0"
format_stamp: "Formatted at 2026-08-10 22:31:12 on dist-test-slave-njxd"
server_key: "ee82f9d053bf371b2a759bbba2e5e900"
server_key_iv: "dfe3a00de5070edfcbcffb69e31a5c1c"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:12.605835 15747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "5401c8c57ca448799399229bb8d1ada0"
format_stamp: "Formatted at 2026-08-10 22:31:12 on dist-test-slave-njxd"
server_key: "ee82f9d053bf371b2a759bbba2e5e900"
server_key_iv: "dfe3a00de5070edfcbcffb69e31a5c1c"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:12.611025 15747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 22:31:12.642000 15766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:12.644088 15747 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.003s sys 0.001s
I20260810 22:31:12.644335 15747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "5401c8c57ca448799399229bb8d1ada0"
format_stamp: "Formatted at 2026-08-10 22:31:12 on dist-test-slave-njxd"
server_key: "ee82f9d053bf371b2a759bbba2e5e900"
server_key_iv: "dfe3a00de5070edfcbcffb69e31a5c1c"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:12.644558 15747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:12.671677 15747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:12.676478 15747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:12.676748 15747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:12.677521 15747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:12.678659 15747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:12.678771 15747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:12.678894 15747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:12.678972 15747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:12.693617 15747 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:34243
I20260810 22:31:12.695051 15747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:31:12.696681 15894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:34243 every 8 connection(s)
I20260810 22:31:12.699875 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 15747
I20260810 22:31:12.700007 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:31:12.700316 1910 external_mini_cluster.cc:1428] Setting key c4a8d3fa79951d31005fb19188cfc32a
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Aug 10 22:31:12 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:12.731652 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:12.699168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40093 (local address 127.1.221.129:55577)
0810 22:31:12.705038 (+ 5870us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:12.705107 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:12.706196 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:12.712981 (+ 6785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:12.712993 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:12.713402 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:12.714229 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:12.714251 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:12.718820 (+ 4569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:12.718824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:12.720287 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:12.720305 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:12.721101 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:12.721105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:12.721325 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:12.722080 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:12.722102 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:12.723880 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:12.726600 (+ 2720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:12.726607 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:12.726624 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:12.726954 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:12.727252 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:12.727255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:12.727257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:12.727419 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:12.729666 (+ 2247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:12.729673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:12.729983 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 22:31:12.730180 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 22:31:12.730506 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5616,"thread_start_us":132,"threads_started":1}
I20260810 22:31:12.732421 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:12.698683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:55577 (local address 127.1.221.190:40093)
0810 22:31:12.712319 (+ 13636us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:12.712332 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:12.712351 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:31:12.712418 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:12.712421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:12.712475 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:12.712602 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:12.714882 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:12.715985 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:12.720443 (+ 4458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:12.720811 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:12.720912 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:12.724079 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:12.724083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:12.724085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:12.724112 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:12.726452 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:12.727066 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:12.727069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:12.727071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:12.727157 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:12.727529 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:12.727532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:12.727534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:12.727746 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:12.727832 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:12.732121 (+ 4289us) server_negotiation.cc:299] Negotiation successful
0810 22:31:12.732252 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13535,"thread_start_us":92,"threads_started":1}
I20260810 22:31:12.735937 15895 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40093
I20260810 22:31:12.736181 15895 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:12.736763 15895 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
I20260810 22:31:12.738528 15656 ts_manager.cc:194] Registered new tserver with Master: 5401c8c57ca448799399229bb8d1ada0 (127.1.221.129:34243)
I20260810 22:31:12.740109 15656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:55577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:31:12.757717 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:41267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40093
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:31:12 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786401072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 22:31:12.922067 15901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:12.922474 15901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:12.922575 15901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:12.928594 15901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:12.928683 15901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:12.928771 15901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:31:12.936663 15901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.221.148:41267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:12.938046 15901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:12.939368 15901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:12.952158 15910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:12.952282 15907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:12.952176 15908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:12.953672 15901 server_base.cc:1034] running on GCE node
I20260810 22:31:12.954190 15901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:12.955013 15901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:12.956208 15901 hybrid_clock.cc:648] HybridClock initialized: now 1786401072956143 us; error 80 us; skew 500 ppm
Aug 10 22:31:12 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:12.959808 15901 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:31:12.960878 15901 webserver.cc:460] Webserver started at http://127.1.221.130:43591/ using document root <none> and password file <none>
I20260810 22:31:12.961620 15901 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:12.961753 15901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:12.962158 15901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:31:12 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:31:12.977248 15901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47"
format_stamp: "Formatted at 2026-08-10 22:31:12 on dist-test-slave-njxd"
server_key: "67a5435548596d7334a54f5277317367"
server_key_iv: "f5e949977266d31c8c40f338fe45b665"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:12.978325 15901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47"
format_stamp: "Formatted at 2026-08-10 22:31:12 on dist-test-slave-njxd"
server_key: "67a5435548596d7334a54f5277317367"
server_key_iv: "f5e949977266d31c8c40f338fe45b665"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:12.990880 15901 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260810 22:31:13.013898 15919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:13.015455 15901 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260810 22:31:13.015626 15901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47"
format_stamp: "Formatted at 2026-08-10 22:31:12 on dist-test-slave-njxd"
server_key: "67a5435548596d7334a54f5277317367"
server_key_iv: "f5e949977266d31c8c40f338fe45b665"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:13.015781 15901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:13.027726 15901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:13.031009 15901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:13.031225 15901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:13.031908 15901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:13.032860 15901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:13.032931 15901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:13.033038 15901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:13.033092 15901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:13.052415 15901 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:43465
I20260810 22:31:13.053748 15901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:31:13.055178 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 15901
I20260810 22:31:13.055307 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:31:13.055656 1910 external_mini_cluster.cc:1428] Setting key 4d8f697f627347591e8f65785d1b594d
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260810 22:31:13.070967 16047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:43465 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Aug 10 22:31:13 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:13.100598 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:13.075000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:33845 (local address 127.1.221.190:40093)
0810 22:31:13.075165 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:13.075171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:13.078371 (+ 3200us) server_negotiation.cc:407] Connection header received
0810 22:31:13.079803 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:13.079808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:13.079875 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:13.079941 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:13.081836 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:13.082641 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:13.084440 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:13.084813 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:13.084900 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:13.090329 (+ 5429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:13.090334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:13.090337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:13.090368 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:13.092219 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:13.097736 (+ 5517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:13.097741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:13.097743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:13.097867 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:13.098419 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:13.098424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:13.098426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:13.098829 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:13.098939 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:13.100111 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0810 22:31:13.100280 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:31:13.102718 16051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:13.073696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40093 (local address 127.1.221.130:33845)
0810 22:31:13.078160 (+ 4464us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:13.078230 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:13.079477 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:13.080449 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:13.080459 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:13.080866 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:13.081639 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:13.081661 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:13.082814 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:13.082819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:13.084276 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:13.084288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:13.085125 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:13.085129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:13.085350 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:13.087606 (+ 2256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:13.087629 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:13.090093 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:13.092844 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:13.092851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:13.092871 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:13.093262 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:13.098008 (+ 4746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:13.098012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:13.098014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:13.098264 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:13.099200 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:13.099209 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:13.099635 (+ 426us) client_negotiation.cc:769] Sending connection context
0810 22:31:13.101041 (+ 1406us) client_negotiation.cc:240] Negotiation successful
0810 22:31:13.101477 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4184,"thread_start_us":193,"threads_started":1}
I20260810 22:31:13.102885 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:41267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40093
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:31:13.104394 16048 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40093
I20260810 22:31:13.104645 16048 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:13.105259 16048 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
I20260810 22:31:13.106658 15656 ts_manager.cc:194] Registered new tserver with Master: c53dcb38b22144c4a4ed3ffd48a6ae47 (127.1.221.130:43465)
I20260810 22:31:13.107422 15656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:33845
Aug 10 22:31:13 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401072, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
W20260810 22:31:13.282311 16053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:13.282675 16053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:13.282732 16053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:13.288623 16053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:13.288694 16053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:13.288785 16053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:31:13.296022 16053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.221.148:41267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:13.297303 16053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:13.298636 16053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:13.310796 16059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:13.311154 16058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:13.311210 16053 server_base.cc:1034] running on GCE node
W20260810 22:31:13.310849 16061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:13.311975 16053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:13.312763 16053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:13.313943 16053 hybrid_clock.cc:648] HybridClock initialized: now 1786401073313927 us; error 58 us; skew 500 ppm
Aug 10 22:31:13 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:13.317866 16053 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:31:13.319090 16053 webserver.cc:460] Webserver started at http://127.1.221.131:38923/ using document root <none> and password file <none>
I20260810 22:31:13.319844 16053 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:13.319959 16053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:13.320220 16053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:31:13 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:31:13.342415 16053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "11eae09eda1e41dd8376365e5e36f07b"
format_stamp: "Formatted at 2026-08-10 22:31:13 on dist-test-slave-njxd"
server_key: "8856fbb22497e8ffe91f977331122488"
server_key_iv: "8c5010539e9aad53452ddb9623963453"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:13.343248 16053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "11eae09eda1e41dd8376365e5e36f07b"
format_stamp: "Formatted at 2026-08-10 22:31:13 on dist-test-slave-njxd"
server_key: "8856fbb22497e8ffe91f977331122488"
server_key_iv: "8c5010539e9aad53452ddb9623963453"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:13.348798 16053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 22:31:13.373155 16069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:13.375023 16053 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260810 22:31:13.375236 16053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "11eae09eda1e41dd8376365e5e36f07b"
format_stamp: "Formatted at 2026-08-10 22:31:13 on dist-test-slave-njxd"
server_key: "8856fbb22497e8ffe91f977331122488"
server_key_iv: "8c5010539e9aad53452ddb9623963453"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:13.375456 16053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:13.412966 16053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:13.417238 16053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:13.417536 16053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:13.418397 16053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:13.419487 16053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:13.419588 16053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:13.419693 16053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:13.419752 16053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:13.433303 16053 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:40171
I20260810 22:31:13.434720 16053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:31:13.435694 16200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:40171 every 8 connection(s)
I20260810 22:31:13.443871 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 16053
I20260810 22:31:13.443992 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:31:13.444362 1910 external_mini_cluster.cc:1428] Setting key a27cd1980ebdc2d5c335bd591b380ea2
Aug 10 22:31:13 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:13.455963 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:13.437994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:35807 (local address 127.1.221.190:40093)
0810 22:31:13.438129 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:13.438133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:13.439115 (+ 982us) server_negotiation.cc:407] Connection header received
0810 22:31:13.440379 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:13.440383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:13.440439 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:13.440492 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:13.442246 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:13.443071 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:13.448096 (+ 5025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:13.448446 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:13.448525 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:13.451790 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:13.451795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:13.451797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:13.451839 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:13.453507 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:13.454167 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:13.454170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:13.454172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:13.454258 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:13.454684 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:13.454687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:13.454689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:13.454912 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:13.454988 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:13.455680 (+ 692us) server_negotiation.cc:299] Negotiation successful
0810 22:31:13.455806 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:31:13.459527 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:13.438467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40093 (local address 127.1.221.131:35807)
0810 22:31:13.438950 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:13.439004 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:13.440116 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:13.440908 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:13.440917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:13.441285 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:13.442026 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:13.442053 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:13.443238 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:13.443241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:13.447948 (+ 4707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:13.447962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:13.448764 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:13.448768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:13.448993 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:13.449719 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:13.449741 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:13.451555 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:13.453683 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:13.453690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:13.453709 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:13.454049 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:13.454360 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:13.454363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:13.454365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:13.454542 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:13.455053 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:13.455059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:13.455387 (+ 328us) client_negotiation.cc:769] Sending connection context
0810 22:31:13.456432 (+ 1045us) client_negotiation.cc:240] Negotiation successful
0810 22:31:13.456796 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":152,"threads_started":1}
I20260810 22:31:13.461063 16202 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40093
I20260810 22:31:13.461305 16202 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:13.461958 16202 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
I20260810 22:31:13.463037 15656 ts_manager.cc:194] Registered new tserver with Master: 11eae09eda1e41dd8376365e5e36f07b (127.1.221.131:40171)
I20260810 22:31:13.463758 15656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:35807
I20260810 22:31:13.470150 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:31:13.470892 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 15747
I20260810 22:31:13.493091 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 15901
I20260810 22:31:13.510171 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 16053
I20260810 22:31:13.520087 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 15618
I20260810 22:31:13.531549 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40093
--webserver_interface=127.1.221.190
--webserver_port=38783
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40093
--ranger_config_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:41267/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:41267/kms with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:13.650291 16206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:13.650574 16206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:13.650650 16206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:13.654784 16206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:31:13.654855 16206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:13.654875 16206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:13.654892 16206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:31:13.654907 16206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:31:13.660739 16206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:41267/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40093
--ranger_config_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=38783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:13.661942 16206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:13.663023 16206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:13.671772 16211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:13.671772 16214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:13.672048 16206 server_base.cc:1034] running on GCE node
W20260810 22:31:13.672029 16216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:13.672662 16206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:13.673852 16206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:13.675045 16206 hybrid_clock.cc:648] HybridClock initialized: now 1786401073675031 us; error 26 us; skew 500 ppm
Aug 10 22:31:13 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:13.677887 16206 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:31:13.678750 16206 webserver.cc:460] Webserver started at http://127.1.221.190:38783/ using document root <none> and password file <none>
I20260810 22:31:13.679271 16206 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:13.679337 16206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 22:31:14 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401073, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:31:14.679553 16206 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:31:14.682168 16226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:14.683566 16206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:31:14.683718 16206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "eaf8b44dd62d4d31bcb5380df19e5f20"
format_stamp: "Formatted at 2026-08-10 22:31:11 on dist-test-slave-njxd"
server_key: "04e749086d703eb378caeaff1e027b2c"
server_key_iv: "1ebdedbdcaceb70bced7821965bd6306"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:14.684298 16206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:14.721344 16206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:14.724548 16206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:14.724761 16206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:14.732198 16206 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:40093
I20260810 22:31:14.732209 16285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:40093 every 8 connection(s)
I20260810 22:31:14.733309 16206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:31:14.737627 16286 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:31:14.738591 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 16206
I20260810 22:31:14.739142 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:41267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:34243
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:40093
--webserver_port=39251
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:41267/kms with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:31:14.739637 16286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20: Bootstrap starting.
I20260810 22:31:14.756753 16286 log.cc:824] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:14.763661 16286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:31:14.764132 16286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20: Bootstrap complete.
I20260810 22:31:14.768508 16286 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40093 } }
I20260810 22:31:14.768915 16286 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf8b44dd62d4d31bcb5380df19e5f20, State: Initialized, Role: FOLLOWER
I20260810 22:31:14.769404 16286 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40093 } }
I20260810 22:31:14.769515 16286 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:31:14.769630 16286 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:31:14.769748 16286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:31:14.771667 16286 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40093 } }
I20260810 22:31:14.772009 16286 leader_election.cc:302] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaf8b44dd62d4d31bcb5380df19e5f20; no voters:
I20260810 22:31:14.772382 16286 leader_election.cc:288] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 22:31:14.772521 16290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:31:14.772763 16290 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [term 2 LEADER]: Becoming Leader. State: Replica: eaf8b44dd62d4d31bcb5380df19e5f20, State: Running, Role: LEADER
I20260810 22:31:14.773156 16290 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40093 } }
I20260810 22:31:14.773684 16286 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:31:14.774610 16291 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40093 } } }
I20260810 22:31:14.774734 16291 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:14.775678 16293 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaf8b44dd62d4d31bcb5380df19e5f20. Latest consensus state: current_term: 2 leader_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf8b44dd62d4d31bcb5380df19e5f20" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40093 } } }
I20260810 22:31:14.775791 16293 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20 [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:14.776037 16301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:31:14.776203 16286 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 22:31:14.782442 16301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:31:14.783357 16301 catalog_manager.cc:1259] Loaded cluster ID: aceab3d8bbba42c694727ce8cee93af6
I20260810 22:31:14.783459 16301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:31:14.786613 16301 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:31:14.787693 16301 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20: Loaded TSK: 0
I20260810 22:31:14.789256 16301 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:31:14.871094 16288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:14.871395 16288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:14.871498 16288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:14.876379 16288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:14.876461 16288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:14.876554 16288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:31:14.882134 16288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:41267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:34243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=39251
--tserver_master_addrs=127.1.221.190:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:14.883306 16288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:14.884732 16288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:14.897459 16328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:14.897475 16331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:14.897748 16329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:14.898458 16288 server_base.cc:1034] running on GCE node
I20260810 22:31:14.898900 16288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:14.899590 16288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:14.900897 16288 hybrid_clock.cc:648] HybridClock initialized: now 1786401074900849 us; error 56 us; skew 500 ppm
Aug 10 22:31:14 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401074, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:14.903973 16288 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:31:14.905001 16288 webserver.cc:460] Webserver started at http://127.1.221.129:39251/ using document root <none> and password file <none>
I20260810 22:31:14.905824 16288 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:14.905936 16288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 22:31:14 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401074, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:31:14.932487 16288 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260810 22:31:14.935902 16341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:14.937656 16288 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 22:31:14.937789 16288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "5401c8c57ca448799399229bb8d1ada0"
format_stamp: "Formatted at 2026-08-10 22:31:12 on dist-test-slave-njxd"
server_key: "ee82f9d053bf371b2a759bbba2e5e900"
server_key_iv: "dfe3a00de5070edfcbcffb69e31a5c1c"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:14.938524 16288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:14.960963 16288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:14.964352 16288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:14.964571 16288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:14.965237 16288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:14.966208 16288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:14.966279 16288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:14.966343 16288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:14.966372 16288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:14.977739 16288 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:34243
I20260810 22:31:14.977973 16467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:34243 every 8 connection(s)
I20260810 22:31:14.978837 16288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:31:14.980027 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 16288
I20260810 22:31:14.980535 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:41267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:43465
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:40093
--webserver_port=43591
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:41267/kms with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:31:14 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401074, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:15.000597 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:14.981429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40093 (local address 127.1.221.129:46831)
0810 22:31:14.983018 (+ 1589us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:14.983073 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:14.983926 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:14.986966 (+ 3040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:14.986986 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:14.987401 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:14.988061 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:14.988082 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:14.989972 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:14.989976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:14.991078 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:14.991089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:14.992104 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:14.992108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:14.992282 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:14.992938 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:14.992957 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:14.994529 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:14.997210 (+ 2681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:14.997215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:14.997227 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:14.997455 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:14.997910 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:14.997913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:14.997915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:14.998091 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:14.998792 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:14.998797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:14.999082 (+ 285us) client_negotiation.cc:769] Sending connection context
0810 22:31:14.999283 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 22:31:14.999583 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1350,"thread_start_us":481,"threads_started":1}
I20260810 22:31:15.002879 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:14.982908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:46831 (local address 127.1.221.190:40093)
0810 22:31:14.985857 (+ 2949us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:14.985883 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:14.985933 (+ 50us) server_negotiation.cc:407] Connection header received
0810 22:31:14.986246 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:14.986258 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:14.986552 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:14.986890 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:14.988272 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:14.989815 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:14.991241 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:14.991935 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:14.993735 (+ 1800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:14.994836 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:14.994842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:14.994858 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:14.994888 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:14.997090 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:14.997643 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:14.997649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:14.997654 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:14.997792 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:14.998230 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:14.998233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:14.998235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:14.998652 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:14.998797 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:15.001691 (+ 2894us) server_negotiation.cc:299] Negotiation successful
0810 22:31:15.002010 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2752,"thread_start_us":116,"threads_started":1}
I20260810 22:31:15.005616 16468 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40093
I20260810 22:31:15.005874 16468 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:15.006412 16468 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
W20260810 22:31:15.008730 16468 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:40093 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 22:31:15.008852 16468 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:15.009019 16468 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
I20260810 22:31:15.010006 16468 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:15.010207 16468 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
W20260810 22:31:15.011036 16468 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:31:15.154145 16471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:15.154528 16471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:15.154613 16471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:15.160848 16471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:15.160936 16471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:15.161031 16471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:31:15.169413 16471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:41267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:43465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=43591
--tserver_master_addrs=127.1.221.190:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:15.170987 16471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:15.172437 16471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:15.185123 16477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:15.185127 16478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:15.185979 16480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:15.186797 16471 server_base.cc:1034] running on GCE node
I20260810 22:31:15.187213 16471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:15.187901 16471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:15.189109 16471 hybrid_clock.cc:648] HybridClock initialized: now 1786401075189085 us; error 42 us; skew 500 ppm
Aug 10 22:31:15 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:15.192080 16471 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:31:15.193063 16471 webserver.cc:460] Webserver started at http://127.1.221.130:43591/ using document root <none> and password file <none>
I20260810 22:31:15.193684 16471 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:15.193763 16471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 22:31:15 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:31:15.214747 16471 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:31:15.217960 16488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:15.219722 16471 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:31:15.219870 16471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47"
format_stamp: "Formatted at 2026-08-10 22:31:12 on dist-test-slave-njxd"
server_key: "67a5435548596d7334a54f5277317367"
server_key_iv: "f5e949977266d31c8c40f338fe45b665"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:15.220630 16471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:15.238730 16471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:15.242553 16471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:15.242794 16471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:15.243554 16471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:15.244668 16471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:15.244732 16471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:15.244788 16471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:15.244820 16471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:15.256206 16471 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:43465
I20260810 22:31:15.256250 16617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:43465 every 8 connection(s)
I20260810 22:31:15.257313 16471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:31:15.266863 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 16471
I20260810 22:31:15.267344 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.221.148:41267/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:40171
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:40093
--webserver_port=38923
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:41267/kms with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:31:15 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:15.281296 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:15.261221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:43051 (local address 127.1.221.190:40093)
0810 22:31:15.261372 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:15.261377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:15.261892 (+ 515us) server_negotiation.cc:407] Connection header received
0810 22:31:15.263230 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:15.263234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:15.263300 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:15.263362 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:15.265159 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:15.265802 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:15.270475 (+ 4673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:15.270857 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:15.270955 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:15.277224 (+ 6269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:15.277228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:15.277231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:15.277272 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:15.278964 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:15.279571 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:15.279575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:15.279579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:15.279672 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:15.280056 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:15.280060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:15.280061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:15.280256 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:15.280366 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:15.280957 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 22:31:15.281110 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:31:15.282302 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:15.260002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40093 (local address 127.1.221.130:43051)
0810 22:31:15.261709 (+ 1707us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:15.261769 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:15.262931 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:15.263827 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:15.263838 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:15.264243 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:15.264971 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:15.264993 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:15.265930 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:15.265933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:15.270274 (+ 4341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:15.270293 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:15.271221 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:15.271226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:15.271469 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:15.272245 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:15.272270 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:15.276990 (+ 4720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:15.279108 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:15.279115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:15.279135 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:15.279450 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:15.279773 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:15.279777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:15.279779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:15.279948 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:15.280366 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:15.280375 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:15.280670 (+ 295us) client_negotiation.cc:769] Sending connection context
0810 22:31:15.280902 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 22:31:15.281263 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1455,"thread_start_us":179,"threads_started":1}
I20260810 22:31:15.283496 16618 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40093
I20260810 22:31:15.283751 16618 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:15.284332 16618 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
W20260810 22:31:15.285955 16618 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:40093 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 22:31:15.286095 16618 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:15.286324 16618 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
I20260810 22:31:15.287222 16618 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:15.287385 16618 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
W20260810 22:31:15.288229 16618 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:31:15.424685 16621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:15.424978 16621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:15.425056 16621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:15.430371 16621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:15.430459 16621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:15.430557 16621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:31:15.437131 16621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.221.148:41267/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:40171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=38923
--tserver_master_addrs=127.1.221.190:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:15.438378 16621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:15.439481 16621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:15.449469 16627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:15.449450 16630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:15.449429 16628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:15.449662 16621 server_base.cc:1034] running on GCE node
I20260810 22:31:15.450341 16621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:15.451117 16621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:15.452333 16621 hybrid_clock.cc:648] HybridClock initialized: now 1786401075452294 us; error 51 us; skew 500 ppm
Aug 10 22:31:15 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:15.456192 16621 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:31:15.457363 16621 webserver.cc:460] Webserver started at http://127.1.221.131:38923/ using document root <none> and password file <none>
I20260810 22:31:15.458144 16621 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:15.458268 16621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 22:31:15 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:31:15.484198 16621 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.001s
I20260810 22:31:15.494947 16641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:15.496807 16621 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260810 22:31:15.497030 16621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "11eae09eda1e41dd8376365e5e36f07b"
format_stamp: "Formatted at 2026-08-10 22:31:13 on dist-test-slave-njxd"
server_key: "8856fbb22497e8ffe91f977331122488"
server_key_iv: "8c5010539e9aad53452ddb9623963453"
server_key_version: "kuduclusterkey@0"
I20260810 22:31:15.497922 16621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:15.530472 16621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:15.534312 16621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:15.534543 16621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:15.535333 16621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:15.536437 16621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:15.536506 16621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:15.536568 16621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:15.536600 16621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:15.548089 16621 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:40171
I20260810 22:31:15.549453 16621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:31:15.550523 16773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:40171 every 8 connection(s)
I20260810 22:31:15.555161 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 16621
Aug 10 22:31:15 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:15.569523 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:15.553905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:44121 (local address 127.1.221.190:40093)
0810 22:31:15.554075 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:15.554080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:15.555856 (+ 1776us) server_negotiation.cc:407] Connection header received
0810 22:31:15.556938 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:15.556942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:15.556998 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:15.557111 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:15.558415 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:15.559245 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:15.560941 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:15.561299 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:15.561400 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:15.564964 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:15.564968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:15.564971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:15.565017 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:15.567062 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:15.567675 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:15.567679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:15.567681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:15.567767 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:15.568178 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:15.568181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:15.568182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:15.568429 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:15.568507 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:15.569210 (+ 703us) server_negotiation.cc:299] Negotiation successful
0810 22:31:15.569335 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"spinlock_wait_cycles":13568}
I20260810 22:31:15.570976 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:15.554420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40093 (local address 127.1.221.131:44121)
0810 22:31:15.555649 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:15.555708 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:15.556698 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:15.557336 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:15.557348 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:15.557762 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:15.558265 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:15.558278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:15.559398 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:15.559402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:15.560802 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:15.560814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:15.561599 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:15.561605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:15.561811 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:15.563079 (+ 1268us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:15.563099 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:15.564780 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:15.567218 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:15.567224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:15.567240 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:15.567560 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:15.567880 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:15.567883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:15.567884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:15.568053 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:15.568596 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:15.568603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:15.568914 (+ 311us) client_negotiation.cc:769] Sending connection context
0810 22:31:15.569797 (+ 883us) client_negotiation.cc:240] Negotiation successful
0810 22:31:15.570104 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"thread_start_us":145,"threads_started":1}
I20260810 22:31:15.572417 16774 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40093
I20260810 22:31:15.572693 16774 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:15.573314 16774 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
I20260810 22:31:15.575055 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:15.556088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.1.221.190:40093)
0810 22:31:15.557471 (+ 1383us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:15.557475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:15.557491 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:31:15.557529 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:15.557532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:15.557607 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:15.557698 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:15.559384 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:15.560107 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:15.562395 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:15.562801 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:15.562901 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:15.564997 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:15.565000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:15.565003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:15.565035 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:15.572139 (+ 7104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:15.572675 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:15.572678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:15.572680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:15.572772 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:15.573080 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:15.573083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:15.573085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:15.573287 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:15.573492 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:15.574708 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0810 22:31:15.574887 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1310,"spinlock_wait_cycles":2432,"thread_start_us":91,"threads_started":1}
W20260810 22:31:15.575135 16774 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:40093 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 22:31:15.575263 16774 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:15.575671 16774 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
I20260810 22:31:15.576668 16774 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:15.576836 16774 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
W20260810 22:31:15.577602 16774 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 22:31:15 dist-test-slave-njxd krb5kdc[15122](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786401075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:16.011260 16468 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:16.011570 16468 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
I20260810 22:31:16.012665 16245 ts_manager.cc:194] Registered new tserver with Master: 5401c8c57ca448799399229bb8d1ada0 (127.1.221.129:34243)
I20260810 22:31:16.014194 16245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:46831
I20260810 22:31:16.288440 16618 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:16.288857 16618 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
I20260810 22:31:16.289839 16245 ts_manager.cc:194] Registered new tserver with Master: c53dcb38b22144c4a4ed3ffd48a6ae47 (127.1.221.130:43465)
I20260810 22:31:16.290462 16245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:43051
Aug 10 22:31:16 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:31:16.577852 16774 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:16.578213 16774 heartbeater.cc:511] Master 127.1.221.190:40093 requested a full tablet report, sending...
I20260810 22:31:16.579283 16245 ts_manager.cc:194] Registered new tserver with Master: 11eae09eda1e41dd8376365e5e36f07b (127.1.221.131:40171)
I20260810 22:31:16.580116 16245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:44121
I20260810 22:31:16.584717 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:31:16.597066 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.587152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.1.221.190:40093)
0810 22:31:16.587466 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:16.587473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:16.587490 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:31:16.587536 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:16.587539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:16.587618 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:16.587718 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:16.588595 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.589440 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.590705 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.591054 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.591155 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:16.592932 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:16.592936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:16.592938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:16.592968 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:16.594858 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:16.595348 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:16.595353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:16.595355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:16.595448 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:16.595731 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:16.595746 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:16.595748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:16.595971 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:16.596100 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:16.596754 (+ 654us) server_negotiation.cc:299] Negotiation successful
0810 22:31:16.596889 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
I20260810 22:31:16.601922 16245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:31:16.607831 16245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 10 22:31:16 dist-test-slave-njxd krb5kdc[15122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401075, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for HTTP/127.1.221.148@KRBTEST.COM
I20260810 22:31:16.649009 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.638034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:40171 (local address 127.0.0.1:34738)
0810 22:31:16.640065 (+ 2031us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:16.640076 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:16.640151 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:16.640965 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:16.640971 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:16.640997 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:16.641147 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:16.641154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.643885 (+ 2731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.643889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:16.646148 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:16.646160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.648366 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.648370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:16.648517 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:16.648567 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 22:31:16.648687 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 22:31:16.648834 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1979,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260810 22:31:16.650309 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.638444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34738 (local address 127.1.221.131:40171)
0810 22:31:16.638803 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:16.638811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:16.640252 (+ 1441us) server_negotiation.cc:407] Connection header received
0810 22:31:16.640349 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:16.640355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:16.640604 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:16.640771 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:16.641289 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.643391 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.646597 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.648013 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.648124 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:16.648301 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:16.649945 (+ 1644us) server_negotiation.cc:299] Negotiation successful
0810 22:31:16.650107 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":131,"threads_started":1}
I20260810 22:31:16.652891 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.636672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:43465 (local address 127.0.0.1:43352)
0810 22:31:16.639785 (+ 3113us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:16.639800 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:16.639889 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:16.646646 (+ 6757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:16.646649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:16.646670 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:16.646761 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:16.646768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.648139 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.648144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:16.649478 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:16.649491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.652463 (+ 2972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.652467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:16.652572 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:16.652601 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 22:31:16.652653 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 22:31:16.652730 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3030,"mutex_wait_us":55,"thread_start_us":136,"threads_started":1}
I20260810 22:31:16.653707 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.636838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.1.221.130:43465)
0810 22:31:16.645834 (+ 8996us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:16.645843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:16.645872 (+ 29us) server_negotiation.cc:407] Connection header received
0810 22:31:16.645965 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:16.645971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:16.646249 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:16.646437 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:16.646908 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.647999 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.649824 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.652312 (+ 2488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.653209 (+ 897us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:16.653377 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:16.653436 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 22:31:16.653536 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8859,"thread_start_us":76,"threads_started":1}
I20260810 22:31:16.654421 16693 tablet_service.cc:1467] Processing CreateTablet for tablet 5efd75987ff54cc698ce3624a0df2136 (DEFAULT_TABLE table=test-table [id=9f3196b5930d48f7a7a4e5bcd03f7ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:31:16.655743 16693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efd75987ff54cc698ce3624a0df2136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:16.656838 16548 tablet_service.cc:1467] Processing CreateTablet for tablet 5efd75987ff54cc698ce3624a0df2136 (DEFAULT_TABLE table=test-table [id=9f3196b5930d48f7a7a4e5bcd03f7ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:31:16.657933 16548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efd75987ff54cc698ce3624a0df2136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:16.659730 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.636628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:34243 (local address 127.0.0.1:53332)
0810 22:31:16.638013 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:16.638085 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:16.638224 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:16.655333 (+ 17109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:16.655339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:16.655368 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:16.655476 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:16.655481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.657031 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.657034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:16.658189 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:16.658202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.659315 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.659319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:16.659421 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:16.659451 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 22:31:16.659504 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 22:31:16.659582 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":756,"threads_started":1}
I20260810 22:31:16.660610 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.636973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53332 (local address 127.1.221.129:34243)
0810 22:31:16.654790 (+ 17817us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:16.654798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:16.654827 (+ 29us) server_negotiation.cc:407] Connection header received
0810 22:31:16.654908 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:16.654914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:16.655135 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:16.655644 (+ 509us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:16.655828 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.656885 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.658373 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.659152 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.660139 (+ 987us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:16.660305 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:16.660363 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 22:31:16.660462 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17691,"thread_start_us":132,"threads_started":1}
I20260810 22:31:16.663677 16400 tablet_service.cc:1467] Processing CreateTablet for tablet 5efd75987ff54cc698ce3624a0df2136 (DEFAULT_TABLE table=test-table [id=9f3196b5930d48f7a7a4e5bcd03f7ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:31:16.664834 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efd75987ff54cc698ce3624a0df2136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:16.667717 16814 tablet_bootstrap.cc:492] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b: Bootstrap starting.
I20260810 22:31:16.670310 16814 tablet_bootstrap.cc:654] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:16.671268 16814 log.cc:824] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:16.673458 16817 tablet_bootstrap.cc:492] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0: Bootstrap starting.
I20260810 22:31:16.673969 16814 tablet_bootstrap.cc:492] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b: No bootstrap required, opened a new log
I20260810 22:31:16.674247 16814 ts_tablet_manager.cc:1397] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 22:31:16.678386 16814 raft_consensus.cc:348] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401c8c57ca448799399229bb8d1ada0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34243 } } peers { permanent_uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43465 } } peers { permanent_uuid: "11eae09eda1e41dd8376365e5e36f07b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40171 } }
I20260810 22:31:16.678737 16817 tablet_bootstrap.cc:654] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:16.678817 16814 raft_consensus.cc:374] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:16.678925 16814 raft_consensus.cc:729] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11eae09eda1e41dd8376365e5e36f07b, State: Initialized, Role: FOLLOWER
I20260810 22:31:16.679483 16814 consensus_queue.cc:260] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401c8c57ca448799399229bb8d1ada0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34243 } } peers { permanent_uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43465 } } peers { permanent_uuid: "11eae09eda1e41dd8376365e5e36f07b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40171 } }
I20260810 22:31:16.680665 16814 ts_tablet_manager.cc:1428] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260810 22:31:16.680565 16817 log.cc:824] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:16.681072 16774 heartbeater.cc:503] Master 127.1.221.190:40093 was elected leader, sending a full tablet report...
I20260810 22:31:16.682614 16815 tablet_bootstrap.cc:492] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47: Bootstrap starting.
I20260810 22:31:16.685123 16817 tablet_bootstrap.cc:492] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0: No bootstrap required, opened a new log
I20260810 22:31:16.685351 16817 ts_tablet_manager.cc:1397] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260810 22:31:16.688239 16815 tablet_bootstrap.cc:654] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:16.690450 16815 log.cc:824] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:16.691121 16817 raft_consensus.cc:348] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401c8c57ca448799399229bb8d1ada0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34243 } } peers { permanent_uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43465 } } peers { permanent_uuid: "11eae09eda1e41dd8376365e5e36f07b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40171 } }
I20260810 22:31:16.691426 16817 raft_consensus.cc:374] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:16.691499 16817 raft_consensus.cc:729] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5401c8c57ca448799399229bb8d1ada0, State: Initialized, Role: FOLLOWER
I20260810 22:31:16.691974 16817 consensus_queue.cc:260] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401c8c57ca448799399229bb8d1ada0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34243 } } peers { permanent_uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43465 } } peers { permanent_uuid: "11eae09eda1e41dd8376365e5e36f07b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40171 } }
I20260810 22:31:16.692615 16468 heartbeater.cc:503] Master 127.1.221.190:40093 was elected leader, sending a full tablet report...
I20260810 22:31:16.693130 16817 ts_tablet_manager.cc:1428] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 22:31:16.693300 16815 tablet_bootstrap.cc:492] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47: No bootstrap required, opened a new log
I20260810 22:31:16.693496 16815 ts_tablet_manager.cc:1397] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260810 22:31:16.700198 16815 raft_consensus.cc:348] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401c8c57ca448799399229bb8d1ada0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34243 } } peers { permanent_uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43465 } } peers { permanent_uuid: "11eae09eda1e41dd8376365e5e36f07b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40171 } }
I20260810 22:31:16.700529 16815 raft_consensus.cc:374] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:16.700637 16815 raft_consensus.cc:729] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c53dcb38b22144c4a4ed3ffd48a6ae47, State: Initialized, Role: FOLLOWER
I20260810 22:31:16.701211 16815 consensus_queue.cc:260] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401c8c57ca448799399229bb8d1ada0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34243 } } peers { permanent_uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43465 } } peers { permanent_uuid: "11eae09eda1e41dd8376365e5e36f07b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40171 } }
I20260810 22:31:16.702409 16815 ts_tablet_manager.cc:1428] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260810 22:31:16.702811 16618 heartbeater.cc:503] Master 127.1.221.190:40093 was elected leader, sending a full tablet report...
I20260810 22:31:16.724275 16822 raft_consensus.cc:482] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:31:16.724593 16822 raft_consensus.cc:504] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401c8c57ca448799399229bb8d1ada0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34243 } } peers { permanent_uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43465 } } peers { permanent_uuid: "11eae09eda1e41dd8376365e5e36f07b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40171 } }
I20260810 22:31:16.727124 16822 leader_election.cc:288] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c53dcb38b22144c4a4ed3ffd48a6ae47 (127.1.221.130:43465), 11eae09eda1e41dd8376365e5e36f07b (127.1.221.131:40171)
I20260810 22:31:16.735955 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.726860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:44299 (local address 127.1.221.130:43465)
0810 22:31:16.727010 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:16.727014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:16.728493 (+ 1479us) server_negotiation.cc:407] Connection header received
0810 22:31:16.728752 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:16.728759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:16.728814 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:16.728865 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:16.729603 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.730415 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.734592 (+ 4177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.735210 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.735336 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:16.735389 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:16.735712 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 22:31:16.735823 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 22:31:16.736289 16469 tablet.cc:2369] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:31:16.736810 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.726946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:50013 (local address 127.1.221.131:40171)
0810 22:31:16.727071 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:16.727076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:16.729036 (+ 1960us) server_negotiation.cc:407] Connection header received
0810 22:31:16.732443 (+ 3407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:16.732447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:16.732508 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:16.732564 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:16.733015 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.733752 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.735240 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.735801 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.735928 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:16.736001 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:16.736599 (+ 598us) server_negotiation.cc:299] Negotiation successful
0810 22:31:16.736690 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:31:16.737279 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.726418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:40171 (local address 127.1.221.129:50013)
0810 22:31:16.728944 (+ 2526us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:16.728959 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:16.730710 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:16.732712 (+ 2002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:16.732715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:16.732733 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:16.732872 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:16.732879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.733904 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.733907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:16.735086 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:16.735113 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.736341 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.736344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:16.736458 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:16.736480 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:31:16.737024 (+ 544us) client_negotiation.cc:240] Negotiation successful
0810 22:31:16.737119 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2180,"thread_start_us":123,"threads_started":1}
I20260810 22:31:16.737979 16721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efd75987ff54cc698ce3624a0df2136" candidate_uuid: "5401c8c57ca448799399229bb8d1ada0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11eae09eda1e41dd8376365e5e36f07b" is_pre_election: true
I20260810 22:31:16.738313 16721 raft_consensus.cc:2454] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5401c8c57ca448799399229bb8d1ada0 in term 0.
I20260810 22:31:16.738571 16826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.726388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:43465 (local address 127.1.221.129:44299)
0810 22:31:16.728390 (+ 2002us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:16.728408 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:16.728586 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:16.729296 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:16.729299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:16.729319 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:16.729464 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:16.729472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.731032 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.731036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:16.734442 (+ 3406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:16.734454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.735461 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.735465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:16.735568 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:16.735592 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:31:16.738282 (+ 2690us) client_negotiation.cc:240] Negotiation successful
0810 22:31:16.738365 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1915,"thread_start_us":147,"threads_started":1}
I20260810 22:31:16.738873 16343 leader_election.cc:302] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11eae09eda1e41dd8376365e5e36f07b, 5401c8c57ca448799399229bb8d1ada0; no voters:
I20260810 22:31:16.739205 16822 raft_consensus.cc:2781] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:31:16.739290 16822 raft_consensus.cc:482] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:31:16.739341 16822 raft_consensus.cc:3037] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:16.739274 16557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efd75987ff54cc698ce3624a0df2136" candidate_uuid: "5401c8c57ca448799399229bb8d1ada0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47" is_pre_election: true
I20260810 22:31:16.739530 16557 raft_consensus.cc:2454] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5401c8c57ca448799399229bb8d1ada0 in term 0.
I20260810 22:31:16.740859 16822 raft_consensus.cc:504] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401c8c57ca448799399229bb8d1ada0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34243 } } peers { permanent_uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43465 } } peers { permanent_uuid: "11eae09eda1e41dd8376365e5e36f07b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40171 } }
I20260810 22:31:16.741276 16822 leader_election.cc:288] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [CANDIDATE]: Term 1 election: Requested vote from peers c53dcb38b22144c4a4ed3ffd48a6ae47 (127.1.221.130:43465), 11eae09eda1e41dd8376365e5e36f07b (127.1.221.131:40171)
I20260810 22:31:16.741657 16557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efd75987ff54cc698ce3624a0df2136" candidate_uuid: "5401c8c57ca448799399229bb8d1ada0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47"
I20260810 22:31:16.741747 16721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efd75987ff54cc698ce3624a0df2136" candidate_uuid: "5401c8c57ca448799399229bb8d1ada0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11eae09eda1e41dd8376365e5e36f07b"
I20260810 22:31:16.741784 16557 raft_consensus.cc:3037] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:16.741868 16721 raft_consensus.cc:3037] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:16.743041 16557 raft_consensus.cc:2454] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5401c8c57ca448799399229bb8d1ada0 in term 1.
I20260810 22:31:16.743129 16721 raft_consensus.cc:2454] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5401c8c57ca448799399229bb8d1ada0 in term 1.
I20260810 22:31:16.743451 16345 leader_election.cc:302] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5401c8c57ca448799399229bb8d1ada0, c53dcb38b22144c4a4ed3ffd48a6ae47; no voters:
I20260810 22:31:16.743714 16822 raft_consensus.cc:2781] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:31:16.743920 16822 raft_consensus.cc:686] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [term 1 LEADER]: Becoming Leader. State: Replica: 5401c8c57ca448799399229bb8d1ada0, State: Running, Role: LEADER
I20260810 22:31:16.744261 16822 consensus_queue.cc:237] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401c8c57ca448799399229bb8d1ada0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34243 } } peers { permanent_uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43465 } } peers { permanent_uuid: "11eae09eda1e41dd8376365e5e36f07b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40171 } }
I20260810 22:31:16.747562 16245 catalog_manager.cc:5495] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5401c8c57ca448799399229bb8d1ada0 (127.1.221.129). New cstate: current_term: 1 leader_uuid: "5401c8c57ca448799399229bb8d1ada0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401c8c57ca448799399229bb8d1ada0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 34243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11eae09eda1e41dd8376365e5e36f07b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40171 } health_report { overall_health: UNKNOWN } } }
W20260810 22:31:16.760897 16619 tablet.cc:2369] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:31:16.780400 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.775295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.1.221.129:34243)
0810 22:31:16.775448 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:16.775454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:16.775472 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:31:16.775568 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:16.775571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:16.775651 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:16.775814 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:31:16.776270 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.776840 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.778124 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:16.778499 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:16.778613 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:16.779063 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:31:16.779670 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:31:16.779801 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:16.780147 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 22:31:16.780218 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:31:16.790319 16721 raft_consensus.cc:1261] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b [term 1 FOLLOWER]: Refusing update from remote peer 5401c8c57ca448799399229bb8d1ada0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:31:16.791275 16827 consensus_queue.cc:1035] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11eae09eda1e41dd8376365e5e36f07b" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:31:16.799293 16557 raft_consensus.cc:1261] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47 [term 1 FOLLOWER]: Refusing update from remote peer 5401c8c57ca448799399229bb8d1ada0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:31:16.800109 16827 consensus_queue.cc:1035] T 5efd75987ff54cc698ce3624a0df2136 P 5401c8c57ca448799399229bb8d1ada0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c53dcb38b22144c4a4ed3ffd48a6ae47" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 43465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:31:16.803768 16776 tablet.cc:2369] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:31:16.812646 16830 mvcc.cc:205] Tried to move back new op lower bound from 7317098810523017216 to 7317098810347180032. Current Snapshot: MvccSnapshot[applied={T|T < 7317098810523017216}]
I20260810 22:31:16.817147 16831 mvcc.cc:205] Tried to move back new op lower bound from 7317098810523017216 to 7317098810347180032. Current Snapshot: MvccSnapshot[applied={T|T < 7317098810523017216}]
I20260810 22:31:16.817636 16835 mvcc.cc:205] Tried to move back new op lower bound from 7317098810523017216 to 7317098810347180032. Current Snapshot: MvccSnapshot[applied={T|T < 7317098810523017216}]
I20260810 22:31:16.823215 16245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:31:16.823388 16245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:31:16.826133 16245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eaf8b44dd62d4d31bcb5380df19e5f20: Sending DeleteTablet for 3 replicas of tablet 5efd75987ff54cc698ce3624a0df2136
I20260810 22:31:16.827260 16400 tablet_service.cc:1514] Processing DeleteTablet for tablet 5efd75987ff54cc698ce3624a0df2136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:16 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:53332
I20260810 22:31:16.827553 16548 tablet_service.cc:1514] Processing DeleteTablet for tablet 5efd75987ff54cc698ce3624a0df2136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:16 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43352
I20260810 22:31:16.828168 16693 tablet_service.cc:1514] Processing DeleteTablet for tablet 5efd75987ff54cc698ce3624a0df2136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:16 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:34738
I20260810 22:31:16.829164 16845 tablet_replica.cc:330] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b: stopping tablet replica
I20260810 22:31:16.829900 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 16288
I20260810 22:31:16.833820 16844 tablet_replica.cc:330] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47: stopping tablet replica
I20260810 22:31:16.834301 16844 raft_consensus.cc:2229] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:31:16.834731 16844 raft_consensus.cc:2258] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:31:16.835965 16845 raft_consensus.cc:2229] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:31:16.836293 16845 raft_consensus.cc:2258] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:31:16.838004 16845 ts_tablet_manager.cc:1905] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:31:16.841077 16845 ts_tablet_manager.cc:1918] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:31:16.841190 16845 log.cc:1196] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/5efd75987ff54cc698ce3624a0df2136
I20260810 22:31:16.841568 16845 ts_tablet_manager.cc:1939] T 5efd75987ff54cc698ce3624a0df2136 P 11eae09eda1e41dd8376365e5e36f07b: Deleting consensus metadata
I20260810 22:31:16.842818 16844 ts_tablet_manager.cc:1905] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:31:16.843040 16228 catalog_manager.cc:4841] TS 11eae09eda1e41dd8376365e5e36f07b (127.1.221.131:40171): tablet 5efd75987ff54cc698ce3624a0df2136 (table test-table [id=9f3196b5930d48f7a7a4e5bcd03f7ff4]) successfully deleted
I20260810 22:31:16.845985 16844 ts_tablet_manager.cc:1918] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:31:16.846168 16844 log.cc:1196] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/5efd75987ff54cc698ce3624a0df2136
I20260810 22:31:16.846668 16844 ts_tablet_manager.cc:1939] T 5efd75987ff54cc698ce3624a0df2136 P c53dcb38b22144c4a4ed3ffd48a6ae47: Deleting consensus metadata
W20260810 22:31:16.847771 16228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:34243 (error 108)
I20260810 22:31:16.847936 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 16471
I20260810 22:31:16.856515 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.856077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:34243 (local address 127.0.0.1:53340)
0810 22:31:16.856325 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:16.856410 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260810 22:31:16.856755 16228 catalog_manager.cc:4584] TS 5401c8c57ca448799399229bb8d1ada0 (127.1.221.129:34243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5efd75987ff54cc698ce3624a0df2136: Network error: Client connection negotiation failed: client connection to 127.1.221.129:34243: connect: Connection refused (error 111)
I20260810 22:31:16.860217 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 16621
I20260810 22:31:16.860951 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:16.860662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:43465 (local address 127.0.0.1:43362)
0810 22:31:16.860828 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:16.860877 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.130:43465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 22:31:16.870060 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 16206
I20260810 22:31:16.879321 1910 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 22:31:17.225513 1910 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 22:31:17.225732 1910 mini_ranger.cc:66] Stopping Ranger...
I20260810 22:31:17.571172 1910 mini_ranger.cc:68] Stopped Ranger
2026-08-10 22:31:17.571 UTC [15153] LOG: received smart shutdown request
2026-08-10 22:31:17.575 UTC [15153] LOG: background worker "logical replication launcher" (PID 15162) exited with exit code 1
2026-08-10 22:31:17.576 UTC [15157] LOG: shutting down
2026-08-10 22:31:17.747 UTC [15153] LOG: database system is shut down
2026-08-10T22:31:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50702 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:31:17 dist-test-slave-njxd krb5kdc[16858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:31:17 dist-test-slave-njxd krb5kdc[16858](info): set up 2 sockets
Aug 10 22:31:17 dist-test-slave-njxd krb5kdc[16858](info): commencing operation
krb5kdc: starting...
W20260810 22:31:19.169937 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.271s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:31:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:31:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:31:19.300933 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39551
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:39551,127.1.221.189:43385,127.1.221.188:39349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:19.415390 16872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:19.415671 16872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:19.415714 16872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:19.419833 16872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:31:19.419905 16872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:19.419925 16872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:19.419941 16872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:31:19.419957 16872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:31:19.425331 16872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:39551,127.1.221.189:43385,127.1.221.188:39349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:39551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:19.426512 16872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:19.427610 16872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:19.436698 16878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:19.436858 16881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:19.436753 16872 server_base.cc:1034] running on GCE node
W20260810 22:31:19.436705 16879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:19.437497 16872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:19.438669 16872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:19.439841 16872 hybrid_clock.cc:648] HybridClock initialized: now 1786401079439821 us; error 32 us; skew 500 ppm
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:19.442795 16872 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:31:19.443804 16872 webserver.cc:460] Webserver started at http://127.1.221.190:36611/ using document root <none> and password file <none>
I20260810 22:31:19.444401 16872 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:19.444464 16872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:19.444697 16872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:31:19.446928 16872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "58f126570dcc4efdbee83414e5a0e84c"
format_stamp: "Formatted at 2026-08-10 22:31:19 on dist-test-slave-njxd"
server_key: "da8118c33582d4bfa1908f3985ccf40f"
server_key_iv: "7eea00c620273732541c30a666479d37"
server_key_version: "encryptionkey@0"
I20260810 22:31:19.447540 16872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "58f126570dcc4efdbee83414e5a0e84c"
format_stamp: "Formatted at 2026-08-10 22:31:19 on dist-test-slave-njxd"
server_key: "da8118c33582d4bfa1908f3985ccf40f"
server_key_iv: "7eea00c620273732541c30a666479d37"
server_key_version: "encryptionkey@0"
I20260810 22:31:19.451520 16872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:31:19.454027 16889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:19.455385 16872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:31:19.455552 16872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "58f126570dcc4efdbee83414e5a0e84c"
format_stamp: "Formatted at 2026-08-10 22:31:19 on dist-test-slave-njxd"
server_key: "da8118c33582d4bfa1908f3985ccf40f"
server_key_iv: "7eea00c620273732541c30a666479d37"
server_key_version: "encryptionkey@0"
I20260810 22:31:19.455678 16872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:19.481961 16872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:19.485569 16872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:19.485776 16872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:19.493854 16872 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:39551
I20260810 22:31:19.493878 16948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:39551 every 8 connection(s)
I20260810 22:31:19.494897 16872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:31:19.497782 16949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:19.498504 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 16872
I20260810 22:31:19.498634 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:31:19.498963 1910 external_mini_cluster.cc:1428] Setting key f0ab32e91fa8fe958bbaa513afe6de25
I20260810 22:31:19.502785 16949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.221.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.189@KRBTEST.COM" created.
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:19.519435 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.505167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47878 (local address 127.1.221.190:39551)
0810 22:31:19.505777 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.505795 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.506403 (+ 608us) server_negotiation.cc:407] Connection header received
0810 22:31:19.507790 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.507794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.507977 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.508132 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:19.509141 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.510212 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.511194 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.511669 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.511844 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.514380 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:19.514386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:19.514397 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:19.514417 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:19.516022 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.516677 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.516683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.516684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.516806 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.517217 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.517219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.517221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.517662 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:19.517785 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:19.518298 (+ 513us) server_negotiation.cc:299] Negotiation successful
0810 22:31:19.518581 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":471,"spinlock_wait_cycles":54528,"thread_start_us":165,"threads_started":1}
I20260810 22:31:19.519439 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.505167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39551 (local address 127.0.0.1:47878)
0810 22:31:19.506216 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.506279 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:19.507462 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:19.508199 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:19.508205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:19.508702 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:19.508990 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.509004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.510346 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.510349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.511067 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.511075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.511851 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.511854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.511970 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.512585 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:19.512601 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:19.514215 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:19.516176 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.516190 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.516207 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.516561 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.516917 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.516924 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.516929 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.517103 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.517788 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:19.517797 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:19.517998 (+ 201us) client_negotiation.cc:769] Sending connection context
0810 22:31:19.518231 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 22:31:19.518674 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":82,"thread_start_us":142,"threads_started":1}
I20260810 22:31:19.522241 16949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:31:19.523036 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.522731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:43385 (local address 127.0.0.1:57230)
0810 22:31:19.522860 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.522949 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:43385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 22:31:19.523267 16893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.189:43385: connect: Connection refused (error 111)
I20260810 22:31:19.524127 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.523929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:43385 (local address 127.0.0.1:57236)
0810 22:31:19.524030 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.524076 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:43385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 22:31:19.524298 16949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:43385: Network error: Client connection negotiation failed: client connection to 127.1.221.189:43385: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.189.keytab.
Entry for principal kudu/127.1.221.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.189.keytab.
I20260810 22:31:19.528803 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:43385
--webserver_interface=127.1.221.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.189.keytab
--principal=kudu/127.1.221.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:39551,127.1.221.189:43385,127.1.221.188:39349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:31:19.573597 16949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } attempt: 1
I20260810 22:31:19.574563 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.574192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:43385 (local address 127.0.0.1:57242)
0810 22:31:19.574373 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.574451 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:43385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 22:31:19.575352 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.575114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:43385 (local address 127.0.0.1:57248)
0810 22:31:19.575227 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.575288 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:43385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 22:31:19.575505 16949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:43385: Network error: Client connection negotiation failed: client connection to 127.1.221.189:43385: connect: Connection refused (error 111)
W20260810 22:31:19.641594 16956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:19.641870 16956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:19.641911 16956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 22:31:19.643697 16949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } attempt: 2
I20260810 22:31:19.644606 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.644270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:43385 (local address 127.0.0.1:57260)
0810 22:31:19.644441 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.644514 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:43385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 22:31:19.645469 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.645231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:43385 (local address 127.0.0.1:57262)
0810 22:31:19.645351 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.645405 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:43385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 22:31:19.645673 16949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:43385: Network error: Client connection negotiation failed: client connection to 127.1.221.189:43385: connect: Connection refused (error 111)
W20260810 22:31:19.646481 16956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:31:19.646533 16956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:19.646574 16956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:19.646610 16956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:31:19.646663 16956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:31:19.652197 16956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:39551,127.1.221.189:43385,127.1.221.188:39349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.189.keytab
--principal=kudu/127.1.221.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:43385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.221.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:19.653329 16956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:19.654479 16956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:19.663311 16967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:19.663309 16964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:19.663316 16962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:19.663579 16956 server_base.cc:1034] running on GCE node
I20260810 22:31:19.664261 16956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:19.665517 16956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:19.666742 16956 hybrid_clock.cc:648] HybridClock initialized: now 1786401079666704 us; error 51 us; skew 500 ppm
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:19.669664 16956 init.cc:377] Logged in from keytab as kudu/127.1.221.189@KRBTEST.COM (short username kudu)
I20260810 22:31:19.670543 16956 webserver.cc:460] Webserver started at http://127.1.221.189:42525/ using document root <none> and password file <none>
I20260810 22:31:19.671073 16956 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:19.671144 16956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:19.671314 16956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:31:19.673362 16956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/data/instance:
uuid: "a06e3e5e0baa437c8d4182a41b7bd62e"
format_stamp: "Formatted at 2026-08-10 22:31:19 on dist-test-slave-njxd"
server_key: "d4caaa094c24c3bb506b861387612bbd"
server_key_iv: "c9c42155dbc9f4f36e94df0978a1e763"
server_key_version: "encryptionkey@0"
I20260810 22:31:19.673954 16956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/wal/instance:
uuid: "a06e3e5e0baa437c8d4182a41b7bd62e"
format_stamp: "Formatted at 2026-08-10 22:31:19 on dist-test-slave-njxd"
server_key: "d4caaa094c24c3bb506b861387612bbd"
server_key_iv: "c9c42155dbc9f4f36e94df0978a1e763"
server_key_version: "encryptionkey@0"
I20260810 22:31:19.677753 16956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:31:19.680106 16974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:19.681326 16956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:31:19.681449 16956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/wal
uuid: "a06e3e5e0baa437c8d4182a41b7bd62e"
format_stamp: "Formatted at 2026-08-10 22:31:19 on dist-test-slave-njxd"
server_key: "d4caaa094c24c3bb506b861387612bbd"
server_key_iv: "c9c42155dbc9f4f36e94df0978a1e763"
server_key_version: "encryptionkey@0"
I20260810 22:31:19.681552 16956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:19.695611 16956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:19.698405 16956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:19.698599 16956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:19.705708 16956 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.189:43385
I20260810 22:31:19.705747 17031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.189:43385 every 8 connection(s)
I20260810 22:31:19.706674 16956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/data/info.pb
I20260810 22:31:19.709395 17033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:19.713130 17033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:31:19.716053 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 16956
I20260810 22:31:19.716181 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-1/wal/instance
I20260810 22:31:19.716470 1910 external_mini_cluster.cc:1428] Setting key fee08023660ee9917a41ac39ad4b0197
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.188@KRBTEST.COM; defaulting to no policy
I20260810 22:31:19.727113 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.715069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47884 (local address 127.1.221.190:39551)
0810 22:31:19.715224 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.715228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.716035 (+ 807us) server_negotiation.cc:407] Connection header received
0810 22:31:19.717080 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.717083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.717141 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.717254 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:19.718454 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.718988 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.720158 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.720410 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.720529 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.723364 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:19.723367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:19.723370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:19.723393 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:19.724898 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.725423 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.725427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.725429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.725515 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.725968 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.725975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.725977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.726179 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:19.726278 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:19.726781 (+ 503us) server_negotiation.cc:299] Negotiation successful
0810 22:31:19.726944 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.188@KRBTEST.COM" created.
I20260810 22:31:19.727982 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.715410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39551 (local address 127.0.0.1:47884)
0810 22:31:19.715870 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.715931 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:19.716871 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:19.717436 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:19.717443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:19.717842 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:19.718309 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.718323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.719140 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.719143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.720028 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.720037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.720582 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.720586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.720755 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.721375 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:19.721391 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:19.723182 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:19.725023 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.725030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.725046 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.725313 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.725624 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.725628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.725631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.725812 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.726295 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:19.726300 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:19.726539 (+ 239us) client_negotiation.cc:769] Sending connection context
0810 22:31:19.726693 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 22:31:19.727010 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":107,"threads_started":1}
I20260810 22:31:19.729436 17033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260810 22:31:19.736888 16949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } attempt: 3
I20260810 22:31:19.740096 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.729983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:43385 (local address 127.0.0.1:57278)
0810 22:31:19.730177 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.730195 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:19.730283 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:19.730795 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:19.730800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:19.731007 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:19.731162 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.731173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.732139 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.732145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.733149 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.733162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.733839 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.733846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.733982 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.734466 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:19.734486 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:19.735896 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:19.738204 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.738209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.738211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.738444 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.738956 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.738959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.738961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.739061 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.739564 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:19.739567 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:19.739650 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 22:31:19.739756 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 22:31:19.739921 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 22:31:19.740803 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.730097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.1.221.189:43385)
0810 22:31:19.730445 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.730455 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.730474 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:31:19.730510 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.730514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.730659 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.730765 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:19.731291 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.732007 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.733276 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.733715 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.733801 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.736043 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:19.736067 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:19.736087 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:19.736117 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:19.738090 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.738597 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.738603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.738605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.738823 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.739187 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.739190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.739192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.739454 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:19.739613 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:19.739954 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 22:31:19.740083 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":168,"threads_started":1}
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260810 22:31:19.743489 17033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.188.keytab.
Entry for principal kudu/127.1.221.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.188.keytab.
I20260810 22:31:19.744329 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.743996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:59944)
0810 22:31:19.744162 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.744247 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260810 22:31:19.744639 16978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
I20260810 22:31:19.745499 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:39349
--webserver_interface=127.1.221.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.188.keytab
--principal=kudu/127.1.221.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:38371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:39551,127.1.221.189:43385,127.1.221.188:39349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:31:19.746069 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.745840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:59950)
0810 22:31:19.745954 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.746003 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 22:31:19.746236 17033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:39349: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
I20260810 22:31:19.747196 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.737453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57284 (local address 127.1.221.189:43385)
0810 22:31:19.737932 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.737940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.737952 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:31:19.737981 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.737983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.738015 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.738118 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:19.739006 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.739693 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.740664 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.741012 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.741123 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.743391 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:19.743395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:19.743398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:19.743422 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:19.745221 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.745846 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.745853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.745857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.745953 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.746320 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.746324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.746326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.746545 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:19.746631 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:19.746977 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 22:31:19.747076 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":135,"threads_started":1}
I20260810 22:31:19.747324 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.737383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:43385 (local address 127.0.0.1:57284)
0810 22:31:19.737522 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.737539 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:19.737680 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:19.738142 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:19.738145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:19.738319 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:19.738883 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.738893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.739837 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.739847 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.740540 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.740550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.741137 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.741141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.741254 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.741785 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:19.741797 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:19.743174 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:19.745431 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.745433 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.745435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.745737 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.746078 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.746084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.746089 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.746193 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.746653 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:19.746659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:19.746795 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 22:31:19.746909 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 22:31:19.747079 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 22:31:19.748142 16949 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:31:19.748937 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.748643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:59956)
0810 22:31:19.748786 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.748850 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 22:31:19.749722 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.749373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:59968)
0810 22:31:19.749518 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.749660 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260810 22:31:19.749876 16949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:39349: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
I20260810 22:31:19.781032 16949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } attempt: 1
I20260810 22:31:19.782050 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.781698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:59980)
0810 22:31:19.781874 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.781954 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 22:31:19.782922 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.782660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:59984)
0810 22:31:19.782798 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.782856 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 22:31:19.783106 16949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:39349: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
I20260810 22:31:19.795513 17033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } attempt: 1
I20260810 22:31:19.796283 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.795971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:60000)
0810 22:31:19.796122 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.796190 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 22:31:19.797021 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.796788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:60012)
0810 22:31:19.796903 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.796957 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 22:31:19.797168 17033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:39349: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
I20260810 22:31:19.858294 16949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } attempt: 2
I20260810 22:31:19.859802 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.858942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:60026)
0810 22:31:19.859641 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.859711 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 22:31:19.860611 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.860377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:60028)
0810 22:31:19.860494 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.860549 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 22:31:19.860796 16949 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:39349: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
W20260810 22:31:19.861902 17043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:19.862205 17043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:19.862299 17043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 22:31:19.865343 17033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } attempt: 2
I20260810 22:31:19.866104 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.865786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:60036)
0810 22:31:19.865948 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.866018 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 22:31:19.866868 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.866638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:60052)
0810 22:31:19.866751 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.866804 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 22:31:19.866948 17043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:31:19.867000 17043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:19.867007 17033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:39349: Network error: Client connection negotiation failed: client connection to 127.1.221.188:39349: connect: Connection refused (error 111)
W20260810 22:31:19.867092 17043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:19.867136 17043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:31:19.867154 17043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:31:19.872736 17043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:39551,127.1.221.189:43385,127.1.221.188:39349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.188.keytab
--principal=kudu/127.1.221.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:39349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.221.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:19.873875 17043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:19.874887 17043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:19.883492 17051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:19.883519 17052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:19.883494 17054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:19.883908 17043 server_base.cc:1034] running on GCE node
I20260810 22:31:19.884441 17043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:19.885617 17043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:19.886788 17043 hybrid_clock.cc:648] HybridClock initialized: now 1786401079886764 us; error 37 us; skew 500 ppm
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:19.889477 17043 init.cc:377] Logged in from keytab as kudu/127.1.221.188@KRBTEST.COM (short username kudu)
I20260810 22:31:19.890332 17043 webserver.cc:460] Webserver started at http://127.1.221.188:34937/ using document root <none> and password file <none>
I20260810 22:31:19.890918 17043 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:19.891010 17043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:19.891260 17043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:31:19.893463 17043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/data/instance:
uuid: "2cf21199cd8c4568a1c1eca2deb40a82"
format_stamp: "Formatted at 2026-08-10 22:31:19 on dist-test-slave-njxd"
server_key: "56f13d724d05787367e739899ece7765"
server_key_iv: "d14131ac243293b8e7f9b1a3a69169f0"
server_key_version: "encryptionkey@0"
I20260810 22:31:19.894112 17043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/wal/instance:
uuid: "2cf21199cd8c4568a1c1eca2deb40a82"
format_stamp: "Formatted at 2026-08-10 22:31:19 on dist-test-slave-njxd"
server_key: "56f13d724d05787367e739899ece7765"
server_key_iv: "d14131ac243293b8e7f9b1a3a69169f0"
server_key_version: "encryptionkey@0"
I20260810 22:31:19.897866 17043 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:31:19.900218 17062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:19.901360 17043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:31:19.901499 17043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/wal
uuid: "2cf21199cd8c4568a1c1eca2deb40a82"
format_stamp: "Formatted at 2026-08-10 22:31:19 on dist-test-slave-njxd"
server_key: "56f13d724d05787367e739899ece7765"
server_key_iv: "d14131ac243293b8e7f9b1a3a69169f0"
server_key_version: "encryptionkey@0"
I20260810 22:31:19.901659 17043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:19.924850 17043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:19.927942 17043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:19.928166 17043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:19.935640 17043 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.188:39349
I20260810 22:31:19.935725 17120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.188:39349 every 8 connection(s)
I20260810 22:31:19.936661 17043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/data/info.pb
I20260810 22:31:19.939656 17121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:19.943897 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 17043
I20260810 22:31:19.944036 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786400913154531-1910-0/minicluster-data/master-2/wal/instance
I20260810 22:31:19.943707 17121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:31:19.944382 1910 external_mini_cluster.cc:1428] Setting key 7cdb1758672f52594dcd13a3b4e45d4f
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260810 22:31:19.958179 17033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } attempt: 3
I20260810 22:31:19.959400 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.946197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.1.221.190:39551)
0810 22:31:19.946371 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.946376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.947076 (+ 700us) server_negotiation.cc:407] Connection header received
0810 22:31:19.948253 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.948257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.948309 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.948398 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:19.949519 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.950330 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.951742 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.952082 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.952194 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.955268 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:19.955275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:19.955281 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:19.955321 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:19.956872 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.957425 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.957427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.957429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.957504 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.957937 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.957943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.957945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.958241 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:19.958359 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:19.959087 (+ 728us) server_negotiation.cc:299] Negotiation successful
0810 22:31:19.959230 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:31:19.960103 16949 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } attempt: 3
I20260810 22:31:19.960968 17124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.946289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.1.221.188:39349)
0810 22:31:19.946851 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.946876 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.947061 (+ 185us) server_negotiation.cc:407] Connection header received
0810 22:31:19.947974 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.947994 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.948257 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.948447 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:19.949358 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.950617 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.951747 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.952323 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.952447 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.955678 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:19.955685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:19.955701 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:19.955728 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:19.957683 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.958215 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.958220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.958223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.958325 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.958697 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.958703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.958706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.959027 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:19.959251 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:19.959499 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 22:31:19.959922 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":366,"spinlock_wait_cycles":6144,"thread_start_us":151,"threads_started":1}
I20260810 22:31:19.960968 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.946289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:39551 (local address 127.0.0.1:47886)
0810 22:31:19.946924 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.946985 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:19.947990 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:19.948478 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:19.948483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:19.948968 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:19.949366 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.949382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.950444 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.950447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.951605 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.951617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.952237 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.952240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.952473 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.953210 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:19.953232 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:19.955003 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:19.956988 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.956995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.957006 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.957322 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.957625 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.957631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.957634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.957832 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.958359 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:19.958367 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:19.958756 (+ 389us) client_negotiation.cc:769] Sending connection context
0810 22:31:19.959625 (+ 869us) client_negotiation.cc:240] Negotiation successful
0810 22:31:19.960697 (+ 1072us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":819,"spinlock_wait_cycles":22784,"thread_start_us":182,"threads_started":1}
I20260810 22:31:19.962518 17121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260810 22:31:19.969259 17035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.958705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:60066)
0810 22:31:19.958934 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.958952 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:19.959035 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:19.959684 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:19.959687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:19.959895 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:19.960050 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.960062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.961835 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.961840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.962504 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.962516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.963163 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.963168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.963282 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.963799 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:19.963815 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:19.965342 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:19.967447 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.967449 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.967452 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.967691 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.968036 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.968041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.968044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.968151 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.968659 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:19.968662 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:19.968765 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 22:31:19.968864 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 22:31:19.969021 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:19.970414 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.958800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60066 (local address 127.1.221.188:39349)
0810 22:31:19.959352 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.959361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.959375 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:31:19.959412 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.959417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.959468 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.960173 (+ 705us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:19.960337 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.961694 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.962671 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.963024 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.963515 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.965497 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:19.965503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:19.965509 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:19.965541 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:19.967344 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.967821 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.967825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.967826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.967922 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.968302 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.968306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.968309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.968546 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:19.969610 (+ 1064us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:19.969704 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 22:31:19.970220 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":123,"threads_started":1}
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260810 22:31:19.972903 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.960488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:60074)
0810 22:31:19.960588 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.960603 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:19.960683 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:19.961983 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:19.961989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:19.962173 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:19.962269 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.962276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.964275 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.964278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.965143 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.965156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.965783 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.965787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.965866 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.966321 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:19.966336 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:19.967819 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:19.971118 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.971122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.971126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.971397 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.971811 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.971815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.971817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.971913 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.972386 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:19.972389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:19.972493 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 22:31:19.972585 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 22:31:19.972714 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 22:31:19.973502 17129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.960576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.1.221.188:39349)
0810 22:31:19.961737 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.961745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.961758 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:31:19.961798 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.961801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.961836 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.962859 (+ 1023us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:19.962995 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.964149 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.965282 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.965647 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.966063 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.967972 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:19.967978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:19.967983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:19.968007 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:19.970103 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.971529 (+ 1426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.971535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.971540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.971629 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.972039 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.972046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.972049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.972271 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:19.973078 (+ 807us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:19.973153 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 22:31:19.973332 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1098,"thread_start_us":100,"threads_started":1}
I20260810 22:31:19.974879 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.963957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.1.221.190:39551)
0810 22:31:19.964299 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.964303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.964314 (+ 11us) server_negotiation.cc:407] Connection header received
0810 22:31:19.964343 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.964345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.964381 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.965167 (+ 786us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:19.965261 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.965957 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.967105 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.967452 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.967910 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.970441 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:19.970446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:19.970451 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:19.970480 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:19.972143 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.972633 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.972636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.972638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.972713 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.973250 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.973254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.973256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.973469 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:19.974506 (+ 1037us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:19.974600 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 22:31:19.974731 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260810 22:31:19.974880 17033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e: Bootstrap starting.
I20260810 22:31:19.975075 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.963478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.1.221.189:43385)
0810 22:31:19.963868 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.963875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.963890 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:31:19.963928 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.963931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.963970 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.964050 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:19.964617 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.965385 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.967378 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.967767 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.967841 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.971285 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:19.971290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:19.971294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:19.971321 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:19.972951 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.973498 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.973502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.973503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.973625 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.973968 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.973971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.973973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.974248 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:19.974354 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:19.974719 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 22:31:19.974846 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291}
I20260810 22:31:19.975219 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.963002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:43385 (local address 127.0.0.1:57288)
0810 22:31:19.963231 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.963246 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:19.963310 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:19.964155 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:19.964158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:19.964331 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:19.964470 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.964481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.965563 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.965568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.967224 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.967239 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.968556 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.968560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.968666 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.969272 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:19.969291 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:19.971095 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:19.973079 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.973082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.973083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.973372 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.973746 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.973749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.973751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.973851 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.974408 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:19.974411 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:19.974503 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 22:31:19.974907 (+ 404us) client_negotiation.cc:240] Negotiation successful
0810 22:31:19.975054 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260810 22:31:19.978066 17033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:19.978336 17121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:31:19.980690 16949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c: Bootstrap starting.
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
Aug 10 22:31:19 dist-test-slave-njxd krb5kdc[16858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401079, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260810 22:31:19.983871 16949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:19.984882 16949 log.cc:824] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:19.986158 17033 log.cc:824] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:19.987774 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.976584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57302 (local address 127.1.221.189:43385)
0810 22:31:19.976683 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.976688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.976816 (+ 128us) server_negotiation.cc:407] Connection header received
0810 22:31:19.976917 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.976919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.976955 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.977016 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:19.977800 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.978523 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.979632 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.979937 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.980017 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.983751 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:19.983754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:19.983756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:19.983785 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:19.985763 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.986302 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.986308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.986310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.986394 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.986749 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.986756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.986760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.986970 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:19.987122 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:19.987512 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 22:31:19.987632 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 22:31:19.987965 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.978787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:39349 (local address 127.0.0.1:60080)
0810 22:31:19.978893 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:19.978906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:19.978980 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:19.979287 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:19.979290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:19.979462 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:19.979537 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.979542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.980228 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.980234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.981006 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:19.981016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.981450 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.981453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:19.981523 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.981960 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:19.981973 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:19.984009 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:19.986356 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.986359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.986362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.986553 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.986875 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:19.986877 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:19.986879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:19.986965 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:19.987350 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:19.987353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:19.987441 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 22:31:19.987547 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 22:31:19.987712 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 22:31:19.988145 16949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c: No bootstrap required, opened a new log
I20260810 22:31:19.988557 17129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.978915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.1.221.188:39349)
0810 22:31:19.979005 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.979012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.979026 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:31:19.979057 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.979060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.979091 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.979184 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:19.979648 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.980123 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.981116 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.981342 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.981450 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:19.984162 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:19.984169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:19.984173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:19.984201 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:19.986194 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.986689 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.986692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.986693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.986775 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:19.987068 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:19.987071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:19.987073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:19.987251 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:19.987359 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:19.987603 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 22:31:19.988425 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":36}
I20260810 22:31:19.991371 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:31:19.991546 17121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82: Bootstrap starting.
I20260810 22:31:19.992293 16949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } }
I20260810 22:31:19.992534 16949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:19.992597 16949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f126570dcc4efdbee83414e5a0e84c, State: Initialized, Role: FOLLOWER
I20260810 22:31:19.993104 16949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } }
I20260810 22:31:19.993674 17137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } } }
I20260810 22:31:19.993840 17137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:31:19.994287 17121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:19.995150 17121 log.cc:824] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:19.995309 17033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e: No bootstrap required, opened a new log
I20260810 22:31:19.997211 17121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82: No bootstrap required, opened a new log
I20260810 22:31:19.998602 16949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:31:19.998839 17033 raft_consensus.cc:348] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } }
I20260810 22:31:19.999070 17033 raft_consensus.cc:374] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:19.999135 17033 raft_consensus.cc:729] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06e3e5e0baa437c8d4182a41b7bd62e, State: Initialized, Role: FOLLOWER
I20260810 22:31:19.999614 17033 consensus_queue.cc:260] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } }
I20260810 22:31:20.002545 17033 sys_catalog.cc:564] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:31:20.003342 17121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } }
I20260810 22:31:20.003610 17121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:20.003679 17121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf21199cd8c4568a1c1eca2deb40a82, State: Initialized, Role: FOLLOWER
I20260810 22:31:20.004235 17121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } }
I20260810 22:31:20.005371 17161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } } }
I20260810 22:31:20.005513 17161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:31:20.005760 17121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:31:20.006699 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.994098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.1.221.190:39551)
0810 22:31:19.994221 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.994225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.994783 (+ 558us) server_negotiation.cc:407] Connection header received
0810 22:31:19.994981 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.994984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.995037 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.995096 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:19.996228 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.996966 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.998217 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:19.998557 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:19.998644 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:20.000675 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:20.000679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:20.000682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:20.000709 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:20.003929 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:20.004729 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:20.004733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:20.004735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:20.004842 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:20.005231 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:20.005234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:20.005236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:20.005480 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:20.005660 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:20.006338 (+ 678us) server_negotiation.cc:299] Negotiation successful
0810 22:31:20.006469 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:31:20.003937 16872 master_runner.cc:186] Error getting master registration for 127.1.221.189:43385: OK, instance_id { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" instance_seqno: 1786401079703711 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 22:31:20.008312 16872 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 22:31:20.011180 17185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:31:20.011345 17185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 22:31:20.015198 17154 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:31:20.015273 17154 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:31:20.015494 17147 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } } }
I20260810 22:31:20.015648 17147 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:31:20.016527 17129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.994818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.1.221.188:39349)
0810 22:31:19.994929 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.994934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:20.000966 (+ 6032us) server_negotiation.cc:407] Connection header received
0810 22:31:20.001163 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:20.001168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:20.001225 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:20.001310 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:20.002082 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:20.002917 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:20.010089 (+ 7172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:20.010422 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:20.010502 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:20.012385 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:20.012389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:20.012395 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:20.012423 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:20.014327 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:20.014846 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:20.014849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:20.014851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:20.014942 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:20.015269 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:20.015273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:20.015275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:20.015506 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:20.015650 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:20.016256 (+ 606us) server_negotiation.cc:299] Negotiation successful
0810 22:31:20.016378 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 22:31:20.017253 17169 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:31:20.017339 17169 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:31:20.017850 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:19.993642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.1.221.189:43385)
0810 22:31:19.993768 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:19.993772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:19.995218 (+ 1446us) server_negotiation.cc:407] Connection header received
0810 22:31:19.995426 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:19.995432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:19.995479 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:19.995532 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:20.008799 (+ 13267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:20.009540 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:20.010638 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:20.010946 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:20.011020 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:20.013133 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:20.013137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:20.013140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:20.013166 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:20.015244 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:20.016052 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:20.016056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:20.016057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:20.016154 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:20.016518 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:20.016521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:20.016523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:20.016757 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:20.016858 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:20.017535 (+ 677us) server_negotiation.cc:299] Negotiation successful
0810 22:31:20.017706 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 22:31:20.018164 16956 master_runner.cc:263] No leader master found from master a06e3e5e0baa437c8d4182a41b7bd62e
I20260810 22:31:20.018230 16956 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 22:31:20.021006 17043 master_runner.cc:263] No leader master found from master 2cf21199cd8c4568a1c1eca2deb40a82
I20260810 22:31:20.021104 17043 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 22:31:20.106530 17137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:31:20.106797 17137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } }
I20260810 22:31:20.108039 17137 leader_election.cc:288] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a06e3e5e0baa437c8d4182a41b7bd62e (127.1.221.189:43385), 2cf21199cd8c4568a1c1eca2deb40a82 (127.1.221.188:39349)
I20260810 22:31:20.108803 17092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58f126570dcc4efdbee83414e5a0e84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" is_pre_election: true
I20260810 22:31:20.108814 17002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58f126570dcc4efdbee83414e5a0e84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" is_pre_election: true
I20260810 22:31:20.109127 17002 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f126570dcc4efdbee83414e5a0e84c in term 0.
I20260810 22:31:20.109131 17092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f126570dcc4efdbee83414e5a0e84c in term 0.
I20260810 22:31:20.109567 16893 leader_election.cc:302] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f126570dcc4efdbee83414e5a0e84c, a06e3e5e0baa437c8d4182a41b7bd62e; no voters:
I20260810 22:31:20.110124 17137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:31:20.110220 17137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:31:20.110265 17137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:20.111625 17137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } }
I20260810 22:31:20.111963 17137 leader_election.cc:288] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [CANDIDATE]: Term 1 election: Requested vote from peers a06e3e5e0baa437c8d4182a41b7bd62e (127.1.221.189:43385), 2cf21199cd8c4568a1c1eca2deb40a82 (127.1.221.188:39349)
I20260810 22:31:20.112594 17002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58f126570dcc4efdbee83414e5a0e84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e"
I20260810 22:31:20.112597 17092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58f126570dcc4efdbee83414e5a0e84c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cf21199cd8c4568a1c1eca2deb40a82"
I20260810 22:31:20.112717 17092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:20.112716 17002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:20.113960 17092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f126570dcc4efdbee83414e5a0e84c in term 1.
I20260810 22:31:20.114353 16893 leader_election.cc:302] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cf21199cd8c4568a1c1eca2deb40a82, 58f126570dcc4efdbee83414e5a0e84c; no voters:
I20260810 22:31:20.114485 17002 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f126570dcc4efdbee83414e5a0e84c in term 1.
I20260810 22:31:20.114651 17137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:31:20.114904 17137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [term 1 LEADER]: Becoming Leader. State: Replica: 58f126570dcc4efdbee83414e5a0e84c, State: Running, Role: LEADER
I20260810 22:31:20.115262 17137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } }
I20260810 22:31:20.117286 17137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58f126570dcc4efdbee83414e5a0e84c. Latest consensus state: current_term: 1 leader_uuid: "58f126570dcc4efdbee83414e5a0e84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } } }
I20260810 22:31:20.117549 17137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:20.119192 17193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:31:20.120246 17193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:31:20.131606 17092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [term 1 FOLLOWER]: Refusing update from remote peer 58f126570dcc4efdbee83414e5a0e84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:31:20.132525 17188 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:31:20.137509 17161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58f126570dcc4efdbee83414e5a0e84c. Latest consensus state: current_term: 1 leader_uuid: "58f126570dcc4efdbee83414e5a0e84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } } }
I20260810 22:31:20.137724 17161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:31:20.139070 17002 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [term 1 FOLLOWER]: Refusing update from remote peer 58f126570dcc4efdbee83414e5a0e84c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:31:20.139838 17188 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:31:20.144089 17147 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58f126570dcc4efdbee83414e5a0e84c. Latest consensus state: current_term: 1 leader_uuid: "58f126570dcc4efdbee83414e5a0e84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } } }
I20260810 22:31:20.144243 17147 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:31:20.152349 17188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58f126570dcc4efdbee83414e5a0e84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } } }
I20260810 22:31:20.152519 17188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:20.154115 17147 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58f126570dcc4efdbee83414e5a0e84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } } }
I20260810 22:31:20.154245 17147 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:31:20.155162 17137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58f126570dcc4efdbee83414e5a0e84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } } }
I20260810 22:31:20.155292 17137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:20.156494 17161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58f126570dcc4efdbee83414e5a0e84c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f126570dcc4efdbee83414e5a0e84c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 39551 } } peers { permanent_uuid: "a06e3e5e0baa437c8d4182a41b7bd62e" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 43385 } } peers { permanent_uuid: "2cf21199cd8c4568a1c1eca2deb40a82" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 39349 } } }
I20260810 22:31:20.156641 17161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:31:20.157209 17194 mvcc.cc:205] Tried to move back new op lower bound from 7317098824198037504 to 7317098824155996160. Current Snapshot: MvccSnapshot[applied={T|T < 7317098824198037504}]
I20260810 22:31:20.157732 17197 mvcc.cc:205] Tried to move back new op lower bound from 7317098824198037504 to 7317098824155996160. Current Snapshot: MvccSnapshot[applied={T|T < 7317098824198037504}]
I20260810 22:31:20.165535 17195 mvcc.cc:205] Tried to move back new op lower bound from 7317098824198037504 to 7317098824155996160. Current Snapshot: MvccSnapshot[applied={T|T < 7317098824198037504}]
I20260810 22:31:20.167377 17193 catalog_manager.cc:1347] Generated new cluster ID: 7f492f25fa954f71a9c6310b4cea09e8
I20260810 22:31:20.167451 17193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:31:20.195569 17193 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:31:20.196964 17193 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:31:20.217186 17193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58f126570dcc4efdbee83414e5a0e84c: Generated new TSK 0
I20260810 22:31:20.218016 17193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:31:21.012722 17185 catalog_manager.cc:1259] Loaded cluster ID: 7f492f25fa954f71a9c6310b4cea09e8
I20260810 22:31:21.012845 17185 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82: loading cluster ID for follower catalog manager: success
I20260810 22:31:21.014537 17185 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82: acquiring CA information for follower catalog manager: success
I20260810 22:31:21.015865 17185 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2cf21199cd8c4568a1c1eca2deb40a82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:31:21.018453 17169 catalog_manager.cc:1259] Loaded cluster ID: 7f492f25fa954f71a9c6310b4cea09e8
I20260810 22:31:21.018518 17169 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e: loading cluster ID for follower catalog manager: success
I20260810 22:31:21.020095 17169 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e: acquiring CA information for follower catalog manager: success
I20260810 22:31:21.021174 17169 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a06e3e5e0baa437c8d4182a41b7bd62e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:31:21.280385 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 16872
I20260810 22:31:21.289364 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 16956
I20260810 22:31:21.297305 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 17043
2026-08-10T22:31:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3423 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:31:21 dist-test-slave-njxd krb5kdc[17210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:31:21 dist-test-slave-njxd krb5kdc[17210](info): set up 2 sockets
Aug 10 22:31:21 dist-test-slave-njxd krb5kdc[17210](info): commencing operation
krb5kdc: starting...
W20260810 22:31:22.407694 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:31:22 dist-test-slave-njxd krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:31:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:31:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:31:22.534579 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:37007
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:36867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:37007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:22.647117 17225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:22.647445 17225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:22.647523 17225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:22.651561 17225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:31:22.651666 17225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:22.651707 17225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:22.651755 17225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:31:22.651798 17225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:31:22.657217 17225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:36867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:37007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:37007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:22.658413 17225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:22.659523 17225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:22.668496 17234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:22.668648 17225 server_base.cc:1034] running on GCE node
W20260810 22:31:22.668503 17232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:22.668723 17231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:22.669425 17225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:22.670549 17225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:22.671741 17225 hybrid_clock.cc:648] HybridClock initialized: now 1786401082671730 us; error 28 us; skew 500 ppm
Aug 10 22:31:22 dist-test-slave-njxd krb5kdc[17210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401082, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:22.674660 17225 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:31:22.675542 17225 webserver.cc:460] Webserver started at http://127.1.221.190:40383/ using document root <none> and password file <none>
I20260810 22:31:22.676117 17225 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:22.676211 17225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:22.676450 17225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:31:22.678601 17225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "6cdab37075184ed480a4c87b8627475a"
format_stamp: "Formatted at 2026-08-10 22:31:22 on dist-test-slave-njxd"
server_key: "d2570d8e470ab470e748a7c4131cd3d7"
server_key_iv: "53179bb54fb02517763d06d78c90ecb0"
server_key_version: "encryptionkey@0"
I20260810 22:31:22.679190 17225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "6cdab37075184ed480a4c87b8627475a"
format_stamp: "Formatted at 2026-08-10 22:31:22 on dist-test-slave-njxd"
server_key: "d2570d8e470ab470e748a7c4131cd3d7"
server_key_iv: "53179bb54fb02517763d06d78c90ecb0"
server_key_version: "encryptionkey@0"
I20260810 22:31:22.683117 17225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:31:22.685636 17241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:22.686975 17225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:31:22.687077 17225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "6cdab37075184ed480a4c87b8627475a"
format_stamp: "Formatted at 2026-08-10 22:31:22 on dist-test-slave-njxd"
server_key: "d2570d8e470ab470e748a7c4131cd3d7"
server_key_iv: "53179bb54fb02517763d06d78c90ecb0"
server_key_version: "encryptionkey@0"
I20260810 22:31:22.687165 17225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:22.698817 17225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:22.701361 17225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:22.701506 17225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:22.708570 17225 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:37007
I20260810 22:31:22.708583 17302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:37007 every 8 connection(s)
I20260810 22:31:22.709514 17225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:31:22.711731 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 17225
I20260810 22:31:22.711874 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:31:22.712193 1910 external_mini_cluster.cc:1428] Setting key f87d27a46d209e5acd628dee3936f9fd
I20260810 22:31:22.712574 17303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:22.718787 17303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a: Bootstrap starting.
Aug 10 22:31:22 dist-test-slave-njxd krb5kdc[17210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:22.721309 17303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:22.722189 17303 log.cc:824] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:22.724653 17303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a: No bootstrap required, opened a new log
I20260810 22:31:22.726729 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:22.713649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.1.221.190:37007)
0810 22:31:22.714100 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:22.714115 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:22.714150 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:31:22.714820 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:22.714837 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:22.715151 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:22.715488 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:22.716327 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:22.717332 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:22.718261 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:22.718747 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:22.718930 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:22.721818 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:22.721823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:22.721838 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:22.721866 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:22.723660 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:22.724197 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:22.724202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:22.724207 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:22.724320 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:22.724643 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:22.724645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:22.724646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:22.725132 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:22.725281 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:22.725658 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 22:31:22.725992 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":147,"threads_started":1}
I20260810 22:31:22.728919 17303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdab37075184ed480a4c87b8627475a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37007 } }
I20260810 22:31:22.729187 17303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:22.729256 17303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdab37075184ed480a4c87b8627475a, State: Initialized, Role: FOLLOWER
I20260810 22:31:22.729763 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:31:22.729750 17303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdab37075184ed480a4c87b8627475a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37007 } }
I20260810 22:31:22.729904 17303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:31:22.729970 17303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:31:22.730062 17303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:22.731112 17303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdab37075184ed480a4c87b8627475a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37007 } }
I20260810 22:31:22.731509 17303 leader_election.cc:302] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cdab37075184ed480a4c87b8627475a; no voters:
I20260810 22:31:22.731860 17303 leader_election.cc:288] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:31:22.731994 17309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:31:22.732278 17309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [term 1 LEADER]: Becoming Leader. State: Replica: 6cdab37075184ed480a4c87b8627475a, State: Running, Role: LEADER
I20260810 22:31:22.732657 17309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdab37075184ed480a4c87b8627475a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37007 } }
I20260810 22:31:22.733158 17303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:31:22.734498 17310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cdab37075184ed480a4c87b8627475a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdab37075184ed480a4c87b8627475a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37007 } } }
I20260810 22:31:22.734499 17311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cdab37075184ed480a4c87b8627475a. Latest consensus state: current_term: 1 leader_uuid: "6cdab37075184ed480a4c87b8627475a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdab37075184ed480a4c87b8627475a" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 37007 } } }
I20260810 22:31:22.734707 17310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:22.734730 17311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:22.735134 17319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:31:22.738039 17319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:31:22.743541 17319 catalog_manager.cc:1347] Generated new cluster ID: fa46d222da664723b2cc3ebdbc7cf507
I20260810 22:31:22.743606 17319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:31:22.763883 17319 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:31:22.765223 17319 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:31:22.771862 17319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cdab37075184ed480a4c87b8627475a: Generated new TSK 0
I20260810 22:31:22.772661 17319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:31:22.785543 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 17225
2026-08-10T22:31:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1487 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:31:22 dist-test-slave-njxd krb5kdc[17333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:31:22 dist-test-slave-njxd krb5kdc[17333](info): set up 2 sockets
Aug 10 22:31:22 dist-test-slave-njxd krb5kdc[17333](info): commencing operation
krb5kdc: starting...
W20260810 22:31:23.866782 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:31:23 dist-test-slave-njxd krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:31:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:31:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:31:23.997092 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:38367
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:24.109288 17347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:24.109629 17347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:24.109725 17347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:24.113692 17347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:31:24.113786 17347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:24.113826 17347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:24.113863 17347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:31:24.113897 17347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:31:24.119060 17347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:24.120116 17347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:24.121153 17347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:24.129609 17355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:24.129683 17354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:24.129621 17357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:24.129886 17347 server_base.cc:1034] running on GCE node
I20260810 22:31:24.130463 17347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:24.131484 17347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:24.132697 17347 hybrid_clock.cc:648] HybridClock initialized: now 1786401084132670 us; error 37 us; skew 500 ppm
Aug 10 22:31:24 dist-test-slave-njxd krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:24.139118 17347 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:31:24.139991 17347 webserver.cc:460] Webserver started at http://127.1.221.190:45201/ using document root <none> and password file <none>
I20260810 22:31:24.140558 17347 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:24.140655 17347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:24.140894 17347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:31:24.143079 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "bb6de92ceedb4043a28d7fb9f1a81258"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "35882b7deedcf070c9984a166cedbc61"
server_key_iv: "f394a95d7df8e120d770d0236ca050fd"
server_key_version: "encryptionkey@0"
I20260810 22:31:24.143668 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "bb6de92ceedb4043a28d7fb9f1a81258"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "35882b7deedcf070c9984a166cedbc61"
server_key_iv: "f394a95d7df8e120d770d0236ca050fd"
server_key_version: "encryptionkey@0"
I20260810 22:31:24.147466 17347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:31:24.149997 17366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:24.151347 17347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:31:24.151513 17347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "bb6de92ceedb4043a28d7fb9f1a81258"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "35882b7deedcf070c9984a166cedbc61"
server_key_iv: "f394a95d7df8e120d770d0236ca050fd"
server_key_version: "encryptionkey@0"
I20260810 22:31:24.151651 17347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:24.184805 17347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:24.187928 17347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:24.188169 17347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:24.195667 17347 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:38367
I20260810 22:31:24.195984 17423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:38367 every 8 connection(s)
I20260810 22:31:24.196661 17347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:31:24.199450 17424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:24.203876 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 17347
I20260810 22:31:24.203982 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:31:24.204283 1910 external_mini_cluster.cc:1428] Setting key 1fa20157c4f6da5ae3b2603c46c7964b
I20260810 22:31:24.205861 17424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Bootstrap starting.
I20260810 22:31:24.208405 17424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:24.209156 17424 log.cc:824] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:24.211408 17424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: No bootstrap required, opened a new log
Aug 10 22:31:24 dist-test-slave-njxd krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:24.215595 17424 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } }
I20260810 22:31:24.215813 17424 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:24.215873 17424 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6de92ceedb4043a28d7fb9f1a81258, State: Initialized, Role: FOLLOWER
I20260810 22:31:24.216334 17424 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } }
I20260810 22:31:24.216459 17424 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:31:24.216557 17424 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:31:24.216667 17424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:24.217849 17424 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } }
I20260810 22:31:24.218235 17424 leader_election.cc:302] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb6de92ceedb4043a28d7fb9f1a81258; no voters:
I20260810 22:31:24.218643 17424 leader_election.cc:288] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:31:24.218734 17429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:31:24.218915 17429 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 1 LEADER]: Becoming Leader. State: Replica: bb6de92ceedb4043a28d7fb9f1a81258, State: Running, Role: LEADER
I20260810 22:31:24.219283 17429 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } }
I20260810 22:31:24.219745 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.205737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35128 (local address 127.1.221.190:38367)
0810 22:31:24.206125 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:24.206150 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:24.206185 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:31:24.206786 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:24.206801 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:24.207130 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:24.207401 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:24.208385 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.209643 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.210582 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.211267 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.211436 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.214362 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:24.214368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:24.214383 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:24.214410 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:24.216395 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:24.216973 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:24.216978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:24.216982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:24.217095 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:24.217395 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:24.217397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:24.217398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:24.217864 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:24.218089 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:24.218329 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 22:31:24.218763 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":138,"threads_started":1}
I20260810 22:31:24.219972 17424 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:31:24.221092 17430 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb6de92ceedb4043a28d7fb9f1a81258. Latest consensus state: current_term: 1 leader_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } } }
I20260810 22:31:24.221139 17431 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } } }
I20260810 22:31:24.221215 17430 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:24.221252 17431 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:24.222141 17441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:31:24.225096 17441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:31:24.230883 17441 catalog_manager.cc:1347] Generated new cluster ID: 5499cecf9fb442ac919ec4f7f82e7b3b
I20260810 22:31:24.230991 17441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:31:24.241464 17441 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:31:24.242792 17441 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:31:24.253630 17441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Generated new TSK 0
I20260810 22:31:24.254578 17441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:31:24.286764 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:38367
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:24.401163 17454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:24.401440 17454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:24.401484 17454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:24.405553 17454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:24.405638 17454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:24.405699 17454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:31:24.411227 17454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:24.412401 17454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:24.413463 17454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:24.423466 17463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:24.423616 17454 server_base.cc:1034] running on GCE node
W20260810 22:31:24.423511 17459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:24.423480 17461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:24.424131 17454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:24.424749 17454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:24.425910 17454 hybrid_clock.cc:648] HybridClock initialized: now 1786401084425896 us; error 26 us; skew 500 ppm
Aug 10 22:31:24 dist-test-slave-njxd krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:24.428683 17454 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:31:24.429647 17454 webserver.cc:460] Webserver started at http://127.1.221.129:41339/ using document root <none> and password file <none>
I20260810 22:31:24.430218 17454 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:24.430310 17454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:24.430552 17454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:31:24.432737 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "1125fc90fdb9408c86e5f352ad51f063"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "323bebdfbfc808a919a465a02a42a684"
server_key_iv: "dff7bef74debf677d84a89b02e96e3e4"
server_key_version: "encryptionkey@0"
I20260810 22:31:24.433318 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "1125fc90fdb9408c86e5f352ad51f063"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "323bebdfbfc808a919a465a02a42a684"
server_key_iv: "dff7bef74debf677d84a89b02e96e3e4"
server_key_version: "encryptionkey@0"
I20260810 22:31:24.437299 17454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:31:24.439815 17470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:24.441149 17454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:31:24.441272 17454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "1125fc90fdb9408c86e5f352ad51f063"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "323bebdfbfc808a919a465a02a42a684"
server_key_iv: "dff7bef74debf677d84a89b02e96e3e4"
server_key_version: "encryptionkey@0"
I20260810 22:31:24.441375 17454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:24.454346 17454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:24.457466 17454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:24.457705 17454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:24.458305 17454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:24.459125 17454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:24.459172 17454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:24.459237 17454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:24.459270 17454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:24.468704 17454 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:33023
I20260810 22:31:24.468732 17598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:33023 every 8 connection(s)
I20260810 22:31:24.469673 17454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:31:24.473255 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 17454
I20260810 22:31:24.473380 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:31:24.473726 1910 external_mini_cluster.cc:1428] Setting key 1811c1f595e22283338e4f8a00688cae
Aug 10 22:31:24 dist-test-slave-njxd krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:24.485018 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.471556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:39839 (local address 127.1.221.190:38367)
0810 22:31:24.471725 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:24.471730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:24.472702 (+ 972us) server_negotiation.cc:407] Connection header received
0810 22:31:24.473695 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:24.473698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:24.473757 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:24.473866 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:24.475274 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.475858 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.477668 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.478081 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.478228 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.480983 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:24.480990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:24.480994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:24.481030 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:24.482847 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:24.483365 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:24.483370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:24.483374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:24.483477 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:24.483808 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:24.483813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:24.483816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:24.484075 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:24.484185 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:24.484682 (+ 497us) server_negotiation.cc:299] Negotiation successful
0810 22:31:24.484821 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260810 22:31:24.485857 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.471931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:39839)
0810 22:31:24.472563 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:24.472603 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:24.473447 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:24.474057 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:24.474066 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:24.474438 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:24.475120 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:24.475138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.475996 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.476002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:24.477469 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:24.477486 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.478352 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.478356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:24.478499 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.479121 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:24.479146 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:24.480803 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:24.482971 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:24.482976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:24.482987 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:24.483252 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:24.483574 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:24.483576 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:24.483577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:24.483710 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:24.484191 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:24.484195 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:24.484426 (+ 231us) client_negotiation.cc:769] Sending connection context
0810 22:31:24.484579 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 22:31:24.484826 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":107,"threads_started":1}
I20260810 22:31:24.487124 17599 heartbeater.cc:347] Connected to a master server at 127.1.221.190:38367
I20260810 22:31:24.487336 17599 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:24.487900 17599 heartbeater.cc:511] Master 127.1.221.190:38367 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260810 22:31:24.489471 17385 ts_manager.cc:194] Registered new tserver with Master: 1125fc90fdb9408c86e5f352ad51f063 (127.1.221.129:33023)
I20260810 22:31:24.490974 17385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:39839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:31:24.504447 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:38367
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:24.615130 17604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:24.615456 17604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:24.615538 17604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:24.619701 17604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:24.619836 17604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:24.619961 17604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:31:24.625564 17604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:24.626701 17604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:24.627776 17604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:24.637507 17613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:24.637643 17604 server_base.cc:1034] running on GCE node
W20260810 22:31:24.637507 17610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:24.637537 17611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:24.638173 17604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:24.638746 17604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:24.639940 17604 hybrid_clock.cc:648] HybridClock initialized: now 1786401084639922 us; error 31 us; skew 500 ppm
Aug 10 22:31:24 dist-test-slave-njxd krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:24.642649 17604 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:31:24.643556 17604 webserver.cc:460] Webserver started at http://127.1.221.130:34473/ using document root <none> and password file <none>
I20260810 22:31:24.644138 17604 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:24.644233 17604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:24.644523 17604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:31:24.646734 17604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "a996534a529b4a55828ec0ecccc8196e"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "f223810009fba1ab3411e2a7504c6345"
server_key_iv: "518bc18ed38260f73100c15e53733fd2"
server_key_version: "encryptionkey@0"
I20260810 22:31:24.647354 17604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "a996534a529b4a55828ec0ecccc8196e"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "f223810009fba1ab3411e2a7504c6345"
server_key_iv: "518bc18ed38260f73100c15e53733fd2"
server_key_version: "encryptionkey@0"
I20260810 22:31:24.651327 17604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:31:24.653854 17621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:24.655160 17604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:31:24.655313 17604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "a996534a529b4a55828ec0ecccc8196e"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "f223810009fba1ab3411e2a7504c6345"
server_key_iv: "518bc18ed38260f73100c15e53733fd2"
server_key_version: "encryptionkey@0"
I20260810 22:31:24.655439 17604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:24.679379 17604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:24.682523 17604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:24.682685 17604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:24.683249 17604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:24.684094 17604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:24.684142 17604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:24.684183 17604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:24.684204 17604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:24.693646 17604 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:36617
I20260810 22:31:24.693675 17751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:36617 every 8 connection(s)
I20260810 22:31:24.694588 17604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:31:24.701634 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 17604
I20260810 22:31:24.701767 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:31:24.702065 1910 external_mini_cluster.cc:1428] Setting key d809ab2a23d18b811e3bc88d7a66496f
Aug 10 22:31:24 dist-test-slave-njxd krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:24.709475 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.696547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:56443 (local address 127.1.221.190:38367)
0810 22:31:24.696692 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:24.696696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:24.697569 (+ 873us) server_negotiation.cc:407] Connection header received
0810 22:31:24.698990 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:24.698993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:24.699054 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:24.699164 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:24.700419 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.700968 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.702364 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.702720 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.702839 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.705555 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:24.705558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:24.705561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:24.705627 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:24.707077 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:24.707734 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:24.707737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:24.707738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:24.707817 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:24.708237 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:24.708239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:24.708240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:24.708413 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:24.708491 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:24.709180 (+ 689us) server_negotiation.cc:299] Negotiation successful
0810 22:31:24.709354 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:31:24.710793 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.696936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:56443)
0810 22:31:24.697406 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:24.697452 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:24.698758 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:24.699355 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:24.699362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:24.699760 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:24.700263 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:24.700276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.701098 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.701103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:24.702240 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:24.702251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.702885 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.702890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:24.703069 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.703723 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:24.703744 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:24.705382 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:24.707206 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:24.707215 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:24.707243 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:24.707625 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:24.707923 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:24.707928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:24.707933 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:24.708134 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:24.708524 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:24.708533 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:24.708888 (+ 355us) client_negotiation.cc:769] Sending connection context
0810 22:31:24.709140 (+ 252us) client_negotiation.cc:240] Negotiation successful
0810 22:31:24.709546 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":121,"threads_started":1}
I20260810 22:31:24.712038 17752 heartbeater.cc:347] Connected to a master server at 127.1.221.190:38367
I20260810 22:31:24.712332 17752 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:24.712955 17752 heartbeater.cc:511] Master 127.1.221.190:38367 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260810 22:31:24.714195 17385 ts_manager.cc:194] Registered new tserver with Master: a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617)
I20260810 22:31:24.714823 17385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:56443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:31:24.732463 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:38367
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:24.847784 17757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:24.848042 17757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:24.848095 17757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:24.852219 17757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:24.852285 17757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:24.852339 17757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:31:24.857789 17757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:24.858844 17757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:24.859967 17757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:24.869860 17764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:24.869931 17767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:24.869889 17765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:24.870090 17757 server_base.cc:1034] running on GCE node
I20260810 22:31:24.870527 17757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:24.871119 17757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:24.872318 17757 hybrid_clock.cc:648] HybridClock initialized: now 1786401084872305 us; error 33 us; skew 500 ppm
Aug 10 22:31:24 dist-test-slave-njxd krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:24.875177 17757 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:31:24.876111 17757 webserver.cc:460] Webserver started at http://127.1.221.131:37031/ using document root <none> and password file <none>
I20260810 22:31:24.876703 17757 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:24.876796 17757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:24.877032 17757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:31:24.879169 17757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "00e00b5b89c2439189e3b582b7087bd0"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "83603b69be2c3b5b491fbcdfa4b717c3"
server_key_iv: "fa20ec8c594e4ee5c6d95204bc153de9"
server_key_version: "encryptionkey@0"
I20260810 22:31:24.879786 17757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "00e00b5b89c2439189e3b582b7087bd0"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "83603b69be2c3b5b491fbcdfa4b717c3"
server_key_iv: "fa20ec8c594e4ee5c6d95204bc153de9"
server_key_version: "encryptionkey@0"
I20260810 22:31:24.883725 17757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 22:31:24.886154 17774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:24.887413 17757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:31:24.887710 17757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "00e00b5b89c2439189e3b582b7087bd0"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "83603b69be2c3b5b491fbcdfa4b717c3"
server_key_iv: "fa20ec8c594e4ee5c6d95204bc153de9"
server_key_version: "encryptionkey@0"
I20260810 22:31:24.887864 17757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:24.916014 17757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:24.919098 17757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:24.919322 17757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:24.920017 17757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:24.920940 17757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:24.921012 17757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:24.921085 17757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:24.921139 17757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:24.930588 17757 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:34951
I20260810 22:31:24.930608 17905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:34951 every 8 connection(s)
I20260810 22:31:24.931520 17757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
Aug 10 22:31:24 dist-test-slave-njxd krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:24.940518 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 17757
I20260810 22:31:24.940717 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:31:24.941030 1910 external_mini_cluster.cc:1428] Setting key a94a114394061171633596f58e9d3de9
I20260810 22:31:24.944972 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.933322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:53211 (local address 127.1.221.190:38367)
0810 22:31:24.933499 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:24.933504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:24.934571 (+ 1067us) server_negotiation.cc:407] Connection header received
0810 22:31:24.935387 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:24.935390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:24.935435 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:24.935508 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:24.936642 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.937164 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.938340 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.938552 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.938632 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.941204 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:24.941207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:24.941209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:24.941231 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:24.942969 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:24.943566 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:24.943569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:24.943571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:24.943673 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:24.943979 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:24.943981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:24.943982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:24.944173 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:24.944248 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:24.944703 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 22:31:24.944799 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 22:31:24.946028 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.933762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:53211)
0810 22:31:24.934417 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:24.934457 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:24.935214 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:24.935671 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:24.935678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:24.936047 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:24.936512 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:24.936524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.937270 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.937273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:24.938239 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:24.938248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.938687 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.938690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:24.938823 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.939437 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:24.939453 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:24.941021 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:24.943085 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:24.943091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:24.943105 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:24.943480 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:24.943764 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:24.943766 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:24.943767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:24.943901 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:24.944264 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:24.944269 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:24.944497 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 22:31:24.944655 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 22:31:24.944918 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":112,"threads_started":1}
I20260810 22:31:24.947109 17906 heartbeater.cc:347] Connected to a master server at 127.1.221.190:38367
I20260810 22:31:24.947359 17906 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:24.947860 17906 heartbeater.cc:511] Master 127.1.221.190:38367 requested a full tablet report, sending...
I20260810 22:31:24.948836 17385 ts_manager.cc:194] Registered new tserver with Master: 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951)
I20260810 22:31:24.949410 17385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:53211
I20260810 22:31:24.955129 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:31:24.965046 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.957175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.1.221.190:38367)
0810 22:31:24.957313 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:24.957318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:24.957478 (+ 160us) server_negotiation.cc:407] Connection header received
0810 22:31:24.957643 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:24.957646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:24.957688 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:24.957754 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:24.958413 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.958974 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.959733 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.959932 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.960050 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.961180 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:24.961183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:24.961184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:24.961204 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:24.963045 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:24.963633 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:24.963637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:24.963639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:24.963737 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:24.964066 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:24.964068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:24.964070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:24.964425 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:24.964580 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:24.964841 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 22:31:24.964936 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:31:24.967669 17385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:31:24.970616 17385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:31:24.984743 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.978736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33023 (local address 127.0.0.1:42632)
0810 22:31:24.979336 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:24.979350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:24.979445 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:24.980531 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:24.980536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:24.980560 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:24.980792 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:24.980802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.981960 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.981966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:24.983371 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:24.983384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.984223 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.984226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:24.984332 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.984368 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 22:31:24.984476 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 22:31:24.984586 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":105,"thread_start_us":92,"threads_started":1}
I20260810 22:31:24.984951 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.979451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.1.221.130:36617)
0810 22:31:24.979823 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:24.979839 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:24.980350 (+ 511us) server_negotiation.cc:407] Connection header received
0810 22:31:24.980480 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:24.980485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:24.980684 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:24.980841 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:24.981151 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.982242 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.983487 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.984291 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.984396 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.984519 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:24.984709 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 22:31:24.984814 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":162,"threads_started":1}
I20260810 22:31:24.984969 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.978676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:34951 (local address 127.0.0.1:50342)
0810 22:31:24.978998 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:24.979049 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:24.979185 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:24.980497 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:24.980503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:24.980522 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:24.980769 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:24.980782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.982276 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.982279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:24.983447 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:24.983457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.984588 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.984591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:24.984694 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.984718 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:31:24.984788 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 22:31:24.984848 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":121,"spinlock_wait_cycles":11520,"thread_start_us":66,"threads_started":1}
I20260810 22:31:24.985103 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.979807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42632 (local address 127.1.221.129:33023)
0810 22:31:24.980147 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:24.980161 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:24.980181 (+ 20us) server_negotiation.cc:407] Connection header received
0810 22:31:24.980247 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:24.980251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:24.980387 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:24.980786 (+ 399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:24.980971 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.981830 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.983518 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.984082 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.984224 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.984385 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:24.984833 (+ 448us) server_negotiation.cc:299] Negotiation successful
0810 22:31:24.984953 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":107,"threads_started":1}
I20260810 22:31:24.985385 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.979017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36617 (local address 127.0.0.1:60878)
0810 22:31:24.980240 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:24.980253 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:24.980367 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:24.980935 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:24.980939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:24.980955 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:24.981033 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:24.981037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.982381 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.982384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:24.983368 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:24.983377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.984466 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.984471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:24.984547 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.984580 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 22:31:24.984651 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 22:31:24.984706 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":24,"thread_start_us":93,"threads_started":1}
I20260810 22:31:24.985515 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:24.979331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.1.221.131:34951)
0810 22:31:24.980012 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:24.980024 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:24.980046 (+ 22us) server_negotiation.cc:407] Connection header received
0810 22:31:24.980126 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:24.980131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:24.980337 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:24.980496 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:24.980918 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.982125 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.983539 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:24.984291 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:24.984396 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:24.984519 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:24.985283 (+ 764us) server_negotiation.cc:299] Negotiation successful
0810 22:31:24.985384 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":111,"threads_started":1}
I20260810 22:31:24.987982 17829 tablet_service.cc:1467] Processing CreateTablet for tablet 8b70e686411943e5b8a4e1ec8753d0c1 (DEFAULT_TABLE table=test-table [id=264a2e6e644c4e7d9db21e682d65b845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:31:24.988152 17524 tablet_service.cc:1467] Processing CreateTablet for tablet 8b70e686411943e5b8a4e1ec8753d0c1 (DEFAULT_TABLE table=test-table [id=264a2e6e644c4e7d9db21e682d65b845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:31:24.988154 17676 tablet_service.cc:1467] Processing CreateTablet for tablet 8b70e686411943e5b8a4e1ec8753d0c1 (DEFAULT_TABLE table=test-table [id=264a2e6e644c4e7d9db21e682d65b845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:31:24.988853 17829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b70e686411943e5b8a4e1ec8753d0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:24.989113 17524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b70e686411943e5b8a4e1ec8753d0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:24.989113 17676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b70e686411943e5b8a4e1ec8753d0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:24.995419 17924 tablet_bootstrap.cc:492] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0: Bootstrap starting.
I20260810 22:31:24.996466 17926 tablet_bootstrap.cc:492] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e: Bootstrap starting.
I20260810 22:31:24.996466 17925 tablet_bootstrap.cc:492] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Bootstrap starting.
I20260810 22:31:24.997998 17924 tablet_bootstrap.cc:654] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:24.998638 17925 tablet_bootstrap.cc:654] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:24.998741 17924 log.cc:824] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:24.999028 17926 tablet_bootstrap.cc:654] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:24.999519 17925 log.cc:824] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:24.999809 17926 log.cc:824] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:25.000770 17924 tablet_bootstrap.cc:492] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0: No bootstrap required, opened a new log
I20260810 22:31:25.000981 17924 ts_tablet_manager.cc:1397] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 22:31:25.001704 17925 tablet_bootstrap.cc:492] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: No bootstrap required, opened a new log
I20260810 22:31:25.001704 17926 tablet_bootstrap.cc:492] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e: No bootstrap required, opened a new log
I20260810 22:31:25.001886 17925 ts_tablet_manager.cc:1397] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:31:25.001886 17926 ts_tablet_manager.cc:1397] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:31:25.003862 17924 raft_consensus.cc:348] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } }
I20260810 22:31:25.004094 17924 raft_consensus.cc:374] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:25.004215 17924 raft_consensus.cc:729] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e00b5b89c2439189e3b582b7087bd0, State: Initialized, Role: FOLLOWER
I20260810 22:31:25.004659 17924 consensus_queue.cc:260] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } }
I20260810 22:31:25.005262 17906 heartbeater.cc:503] Master 127.1.221.190:38367 was elected leader, sending a full tablet report...
I20260810 22:31:25.005725 17926 raft_consensus.cc:348] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } }
I20260810 22:31:25.005993 17926 raft_consensus.cc:374] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:25.006083 17926 raft_consensus.cc:729] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a996534a529b4a55828ec0ecccc8196e, State: Initialized, Role: FOLLOWER
I20260810 22:31:25.006083 17925 raft_consensus.cc:348] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } }
I20260810 22:31:25.006349 17925 raft_consensus.cc:374] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:25.006434 17925 raft_consensus.cc:729] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1125fc90fdb9408c86e5f352ad51f063, State: Initialized, Role: FOLLOWER
I20260810 22:31:25.006553 17924 ts_tablet_manager.cc:1428] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260810 22:31:25.006572 17926 consensus_queue.cc:260] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } }
I20260810 22:31:25.006994 17925 consensus_queue.cc:260] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } }
I20260810 22:31:25.007685 17599 heartbeater.cc:503] Master 127.1.221.190:38367 was elected leader, sending a full tablet report...
I20260810 22:31:25.008077 17926 ts_tablet_manager.cc:1428] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:31:25.008402 17752 heartbeater.cc:503] Master 127.1.221.190:38367 was elected leader, sending a full tablet report...
I20260810 22:31:25.008889 17925 ts_tablet_manager.cc:1428] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 22:31:25.051064 17934 raft_consensus.cc:482] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:31:25.051388 17934 raft_consensus.cc:504] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } }
I20260810 22:31:25.052577 17934 leader_election.cc:288] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1125fc90fdb9408c86e5f352ad51f063 (127.1.221.129:33023), a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617)
I20260810 22:31:25.056949 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.052864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36617 (local address 127.1.221.131:36891)
0810 22:31:25.053026 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:25.053063 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:25.053178 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:25.053450 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:25.053454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:25.053482 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:25.053653 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.053661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.054532 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.054536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.055521 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.055535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.056466 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.056469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.056582 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.056608 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:31:25.056691 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 22:31:25.056789 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":1}
I20260810 22:31:25.056951 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.053018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:36891 (local address 127.1.221.130:36617)
0810 22:31:25.053183 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:25.053191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:25.053211 (+ 20us) server_negotiation.cc:407] Connection header received
0810 22:31:25.053276 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:25.053279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:25.053344 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:25.053416 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:25.053773 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.054432 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.055666 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.056330 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.056481 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.056539 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:25.056725 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 22:31:25.056806 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 22:31:25.057488 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.052864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33023 (local address 127.1.221.131:49967)
0810 22:31:25.053450 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:25.053465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:25.053556 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:25.053847 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:25.053849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:25.053868 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:25.053969 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.053975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.054944 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.054948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.056106 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.056118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.056994 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.057001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.057099 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.057141 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 22:31:25.057269 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 22:31:25.057340 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":137,"thread_start_us":166,"threads_started":1}
I20260810 22:31:25.057539 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.053018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:49967 (local address 127.1.221.129:33023)
0810 22:31:25.053165 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:25.053170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:25.053558 (+ 388us) server_negotiation.cc:407] Connection header received
0810 22:31:25.053669 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:25.053671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:25.053734 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:25.053819 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:25.054084 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.054838 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.056245 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.056850 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.056969 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.057031 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:25.057320 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 22:31:25.057400 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 22:31:25.057834 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b70e686411943e5b8a4e1ec8753d0c1" candidate_uuid: "00e00b5b89c2439189e3b582b7087bd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a996534a529b4a55828ec0ecccc8196e" is_pre_election: true
I20260810 22:31:25.058168 17546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b70e686411943e5b8a4e1ec8753d0c1" candidate_uuid: "00e00b5b89c2439189e3b582b7087bd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1125fc90fdb9408c86e5f352ad51f063" is_pre_election: true
I20260810 22:31:25.058203 17699 raft_consensus.cc:2454] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00e00b5b89c2439189e3b582b7087bd0 in term 0.
I20260810 22:31:25.058455 17546 raft_consensus.cc:2454] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00e00b5b89c2439189e3b582b7087bd0 in term 0.
I20260810 22:31:25.058825 17777 leader_election.cc:302] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e00b5b89c2439189e3b582b7087bd0, a996534a529b4a55828ec0ecccc8196e; no voters:
I20260810 22:31:25.059136 17934 raft_consensus.cc:2781] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:31:25.059226 17934 raft_consensus.cc:482] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:31:25.059271 17934 raft_consensus.cc:3037] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:25.060348 17934 raft_consensus.cc:504] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } }
I20260810 22:31:25.060698 17934 leader_election.cc:288] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [CANDIDATE]: Term 1 election: Requested vote from peers 1125fc90fdb9408c86e5f352ad51f063 (127.1.221.129:33023), a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617)
I20260810 22:31:25.061051 17699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b70e686411943e5b8a4e1ec8753d0c1" candidate_uuid: "00e00b5b89c2439189e3b582b7087bd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a996534a529b4a55828ec0ecccc8196e"
I20260810 22:31:25.061051 17546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b70e686411943e5b8a4e1ec8753d0c1" candidate_uuid: "00e00b5b89c2439189e3b582b7087bd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1125fc90fdb9408c86e5f352ad51f063"
I20260810 22:31:25.061184 17699 raft_consensus.cc:3037] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:25.061213 17546 raft_consensus.cc:3037] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:25.062584 17546 raft_consensus.cc:2454] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00e00b5b89c2439189e3b582b7087bd0 in term 1.
I20260810 22:31:25.062584 17699 raft_consensus.cc:2454] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00e00b5b89c2439189e3b582b7087bd0 in term 1.
I20260810 22:31:25.062912 17778 leader_election.cc:302] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e00b5b89c2439189e3b582b7087bd0, 1125fc90fdb9408c86e5f352ad51f063; no voters:
I20260810 22:31:25.063162 17934 raft_consensus.cc:2781] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:31:25.063408 17934 raft_consensus.cc:686] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [term 1 LEADER]: Becoming Leader. State: Replica: 00e00b5b89c2439189e3b582b7087bd0, State: Running, Role: LEADER
I20260810 22:31:25.063761 17934 consensus_queue.cc:237] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } }
I20260810 22:31:25.066996 17384 catalog_manager.cc:5495] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131). New cstate: current_term: 1 leader_uuid: "00e00b5b89c2439189e3b582b7087bd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } health_report { overall_health: UNKNOWN } } }
I20260810 22:31:25.082135 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.079038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.1.221.131:34951)
0810 22:31:25.079165 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:25.079173 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:25.079199 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:31:25.079261 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:25.079263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:25.079310 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:25.079385 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:31:25.079592 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.080070 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.080881 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.081073 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.081144 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.081388 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:31:25.081823 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:31:25.081912 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:25.081988 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 22:31:25.082033 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:31:25.087839 17546 raft_consensus.cc:1261] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Refusing update from remote peer 00e00b5b89c2439189e3b582b7087bd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:31:25.087837 17699 raft_consensus.cc:1261] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e [term 1 FOLLOWER]: Refusing update from remote peer 00e00b5b89c2439189e3b582b7087bd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:31:25.088572 17934 consensus_queue.cc:1035] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:31:25.088891 17940 consensus_queue.cc:1035] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:31:25.096596 17944 mvcc.cc:205] Tried to move back new op lower bound from 7317098844512452608 to 7317098844423843840. Current Snapshot: MvccSnapshot[applied={T|T < 7317098844512452608}]
I20260810 22:31:25.097095 17945 mvcc.cc:205] Tried to move back new op lower bound from 7317098844512452608 to 7317098844423843840. Current Snapshot: MvccSnapshot[applied={T|T < 7317098844512452608}]
I20260810 22:31:25.104281 17383 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:31:25.104444 17383 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:31:25.106640 17383 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Sending DeleteTablet for 3 replicas of tablet 8b70e686411943e5b8a4e1ec8753d0c1
I20260810 22:31:25.107357 17829 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b70e686411943e5b8a4e1ec8753d0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:25 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:50342
I20260810 22:31:25.107439 17524 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b70e686411943e5b8a4e1ec8753d0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:25 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:42632
I20260810 22:31:25.107555 17676 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b70e686411943e5b8a4e1ec8753d0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:25 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:60878
I20260810 22:31:25.107971 17954 tablet_replica.cc:330] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: stopping tablet replica
I20260810 22:31:25.107975 17955 tablet_replica.cc:330] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e: stopping tablet replica
I20260810 22:31:25.108217 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 17454
I20260810 22:31:25.108292 17955 raft_consensus.cc:2229] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:31:25.110183 17955 raft_consensus.cc:2258] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:31:25.115965 17955 ts_tablet_manager.cc:1905] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:31:25.117848 17953 tablet_replica.cc:330] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0: stopping tablet replica
I20260810 22:31:25.118285 17953 raft_consensus.cc:2229] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:31:25.118816 17955 ts_tablet_manager.cc:1918] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:31:25.118829 17953 raft_consensus.cc:2258] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:31:25.118988 17955 log.cc:1196] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/8b70e686411943e5b8a4e1ec8753d0c1
I20260810 22:31:25.119410 17955 ts_tablet_manager.cc:1939] T 8b70e686411943e5b8a4e1ec8753d0c1 P a996534a529b4a55828ec0ecccc8196e: Deleting consensus metadata
I20260810 22:31:25.120574 17953 ts_tablet_manager.cc:1905] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:31:25.120857 17369 catalog_manager.cc:4841] TS a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617): tablet 8b70e686411943e5b8a4e1ec8753d0c1 (table test-table [id=264a2e6e644c4e7d9db21e682d65b845]) successfully deleted
W20260810 22:31:25.121600 17370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:33023 (error 108)
I20260810 22:31:25.122489 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 17604
I20260810 22:31:25.122656 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.122340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33023 (local address 127.0.0.1:42640)
0810 22:31:25.122476 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:25.122553 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 22:31:25.127879 17370 catalog_manager.cc:4584] TS 1125fc90fdb9408c86e5f352ad51f063 (127.1.221.129:33023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b70e686411943e5b8a4e1ec8753d0c1: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33023: connect: Connection refused (error 111)
I20260810 22:31:25.129518 17953 ts_tablet_manager.cc:1918] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:31:25.129698 17953 log.cc:1196] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/8b70e686411943e5b8a4e1ec8753d0c1
I20260810 22:31:25.130103 17953 ts_tablet_manager.cc:1939] T 8b70e686411943e5b8a4e1ec8753d0c1 P 00e00b5b89c2439189e3b582b7087bd0: Deleting consensus metadata
I20260810 22:31:25.131405 17367 catalog_manager.cc:4841] TS 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951): tablet 8b70e686411943e5b8a4e1ec8753d0c1 (table test-table [id=264a2e6e644c4e7d9db21e682d65b845]) successfully deleted
I20260810 22:31:25.135545 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 17757
I20260810 22:31:25.142969 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 17347
I20260810 22:31:25.152628 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:38367
--webserver_interface=127.1.221.190
--webserver_port=45201
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:25.264086 17958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:25.264397 17958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:25.264480 17958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:25.268450 17958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:31:25.268540 17958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:25.268579 17958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:25.268638 17958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:31:25.268675 17958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:31:25.273900 17958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=45201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:25.275022 17958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:25.276090 17958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:25.290092 17967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:25.290092 17969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:25.290159 17966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:25.290488 17958 server_base.cc:1034] running on GCE node
I20260810 22:31:25.291109 17958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:25.292325 17958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:25.293483 17958 hybrid_clock.cc:648] HybridClock initialized: now 1786401085293468 us; error 25 us; skew 500 ppm
Aug 10 22:31:25 dist-test-slave-njxd krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401085, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:25.296453 17958 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:31:25.297477 17958 webserver.cc:460] Webserver started at http://127.1.221.190:45201/ using document root <none> and password file <none>
I20260810 22:31:25.298097 17958 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:25.298206 17958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:25.302220 17958 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:31:25.304471 17976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:25.305737 17958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:31:25.305868 17958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "bb6de92ceedb4043a28d7fb9f1a81258"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "35882b7deedcf070c9984a166cedbc61"
server_key_iv: "f394a95d7df8e120d770d0236ca050fd"
server_key_version: "encryptionkey@0"
I20260810 22:31:25.306485 17958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:25.322947 17958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:25.326498 17958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:25.326669 17958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:25.334172 17958 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:38367
I20260810 22:31:25.334494 18033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:38367 every 8 connection(s)
I20260810 22:31:25.335345 17958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:31:25.339576 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 17958
I20260810 22:31:25.339841 18036 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:31:25.340067 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:33023
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:38367
--webserver_port=41339
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:31:25.341711 18036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Bootstrap starting.
I20260810 22:31:25.360244 18036 log.cc:824] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:25.368914 18036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:31:25.369311 18036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Bootstrap complete.
I20260810 22:31:25.373708 18036 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } }
I20260810 22:31:25.374147 18036 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6de92ceedb4043a28d7fb9f1a81258, State: Initialized, Role: FOLLOWER
I20260810 22:31:25.374653 18036 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } }
I20260810 22:31:25.374799 18036 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:31:25.374894 18036 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:31:25.375016 18036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:31:25.376915 18036 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } }
I20260810 22:31:25.377257 18036 leader_election.cc:302] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb6de92ceedb4043a28d7fb9f1a81258; no voters:
I20260810 22:31:25.377645 18036 leader_election.cc:288] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 22:31:25.377748 18040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:31:25.377966 18040 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 2 LEADER]: Becoming Leader. State: Replica: bb6de92ceedb4043a28d7fb9f1a81258, State: Running, Role: LEADER
I20260810 22:31:25.378324 18040 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } }
I20260810 22:31:25.378934 18036 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:31:25.379853 18042 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } } }
I20260810 22:31:25.379997 18042 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:25.380354 18051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:31:25.380359 18043 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb6de92ceedb4043a28d7fb9f1a81258. Latest consensus state: current_term: 2 leader_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } } }
I20260810 22:31:25.380496 18043 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:25.385303 18051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:31:25.386076 18051 catalog_manager.cc:1259] Loaded cluster ID: 5499cecf9fb442ac919ec4f7f82e7b3b
I20260810 22:31:25.386159 18051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:31:25.387295 18051 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 22:31:25.390070 18051 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 22:31:25.394587 18051 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:31:25.395316 18051 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Loaded TSK: 0
I20260810 22:31:25.396275 18051 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:31:25.464941 18038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:25.465193 18038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:25.465240 18038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:25.469264 18038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:25.469352 18038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:25.469434 18038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:31:25.474746 18038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:33023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=41339
--tserver_master_addrs=127.1.221.190:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:25.475744 18038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:25.476737 18038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:25.487600 18072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:25.487648 18070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:25.487820 18068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:25.488102 18038 server_base.cc:1034] running on GCE node
I20260810 22:31:25.488519 18038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:25.489215 18038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:25.490401 18038 hybrid_clock.cc:648] HybridClock initialized: now 1786401085490372 us; error 42 us; skew 500 ppm
Aug 10 22:31:25 dist-test-slave-njxd krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401085, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:25.493340 18038 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:31:25.494290 18038 webserver.cc:460] Webserver started at http://127.1.221.129:41339/ using document root <none> and password file <none>
I20260810 22:31:25.494871 18038 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:25.494987 18038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:25.499114 18038 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 22:31:25.501398 18085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:25.502758 18038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:31:25.502887 18038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "1125fc90fdb9408c86e5f352ad51f063"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "323bebdfbfc808a919a465a02a42a684"
server_key_iv: "dff7bef74debf677d84a89b02e96e3e4"
server_key_version: "encryptionkey@0"
I20260810 22:31:25.503438 18038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:25.532331 18038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:25.535796 18038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:25.535949 18038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:25.536614 18038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:25.537748 18092 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 22:31:25.541888 18038 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 22:31:25.541962 18038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260810 22:31:25.542064 18038 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 22:31:25.543741 18038 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 22:31:25.543812 18038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 22:31:25.544070 18092 tablet_bootstrap.cc:492] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Bootstrap starting.
I20260810 22:31:25.554972 18038 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:33023
I20260810 22:31:25.555070 18213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:33023 every 8 connection(s)
I20260810 22:31:25.556073 18038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:31:25.557883 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 18038
I20260810 22:31:25.558357 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:36617
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:38367
--webserver_port=34473
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:31:25.564328 18092 log.cc:824] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:31:25 dist-test-slave-njxd krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401085, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:25.574295 18092 tablet_bootstrap.cc:492] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:31:25.574791 18092 tablet_bootstrap.cc:492] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Bootstrap complete.
I20260810 22:31:25.575222 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.558349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:34811 (local address 127.1.221.190:38367)
0810 22:31:25.559253 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:25.559288 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:25.560795 (+ 1507us) server_negotiation.cc:407] Connection header received
0810 22:31:25.562275 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:25.562292 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:25.562731 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:25.563137 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:25.564311 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.565761 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.566964 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.567463 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.567645 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.570188 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:25.570193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:25.570205 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:25.570231 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:25.571959 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:25.572513 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:25.572518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:25.572522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:25.572638 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:25.572995 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:25.572999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:25.573001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:25.573340 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:25.573488 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:25.574089 (+ 601us) server_negotiation.cc:299] Negotiation successful
0810 22:31:25.574357 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":132,"threads_started":1}
I20260810 22:31:25.575234 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.558338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:34811)
0810 22:31:25.560568 (+ 2230us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:25.560633 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:25.561808 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:25.563183 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:25.563189 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:25.563577 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:25.564118 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.564132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.565932 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.565935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.566828 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.566837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.567615 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.567619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.567822 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.568448 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:25.568464 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:25.569988 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:25.572091 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:25.572097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:25.572111 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:25.572378 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:25.572743 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:25.572746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:25.572748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:25.572893 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:25.573460 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:25.573466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:25.573779 (+ 313us) client_negotiation.cc:769] Sending connection context
0810 22:31:25.573959 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 22:31:25.574296 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1974,"thread_start_us":156,"threads_started":1}
I20260810 22:31:25.576463 18092 ts_tablet_manager.cc:1397] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.008s
I20260810 22:31:25.577801 18214 heartbeater.cc:347] Connected to a master server at 127.1.221.190:38367
I20260810 22:31:25.578068 18214 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:25.578603 18214 heartbeater.cc:511] Master 127.1.221.190:38367 requested a full tablet report, sending...
I20260810 22:31:25.579459 18092 raft_consensus.cc:348] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } }
I20260810 22:31:25.580016 18092 raft_consensus.cc:729] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1125fc90fdb9408c86e5f352ad51f063, State: Initialized, Role: FOLLOWER
I20260810 22:31:25.580549 18092 consensus_queue.cc:260] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } }
I20260810 22:31:25.581310 17995 ts_manager.cc:194] Registered new tserver with Master: 1125fc90fdb9408c86e5f352ad51f063 (127.1.221.129:33023)
I20260810 22:31:25.581688 18092 ts_tablet_manager.cc:1428] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 22:31:25.584676 17995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:34811
I20260810 22:31:25.587329 18214 heartbeater.cc:503] Master 127.1.221.190:38367 was elected leader, sending a full tablet report...
Aug 10 22:31:25 dist-test-slave-njxd krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401085, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260810 22:31:25.595036 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.584026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33023 (local address 127.0.0.1:42654)
0810 22:31:25.584742 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:25.584796 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:25.584935 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:25.585457 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:25.585464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:25.585924 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:25.586089 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.586097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.587418 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.587421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.588350 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.588362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.589115 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.589121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.589246 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.589837 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:25.589854 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:25.591322 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:25.593250 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:25.593261 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:25.593266 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:25.593496 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:25.593837 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:25.593840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:25.593842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:25.593930 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:25.594380 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:25.594387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:25.594509 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 22:31:25.594660 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 22:31:25.594857 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":90,"threads_started":1}
I20260810 22:31:25.595052 18222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.584128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.1.221.129:33023)
0810 22:31:25.584469 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:25.584479 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:25.584883 (+ 404us) server_negotiation.cc:407] Connection header received
0810 22:31:25.585083 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:25.585088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:25.585289 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:25.585441 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:25.586218 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.587282 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.588484 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.588984 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.589079 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.591461 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:25.591470 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:25.591482 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:25.591513 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:25.593134 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:25.593641 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:25.593654 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:25.593655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:25.593730 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:25.594030 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:25.594034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:25.594036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:25.594252 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:25.594403 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:25.594756 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 22:31:25.594909 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":260,"thread_start_us":102,"threads_started":1}
I20260810 22:31:25.595922 18138 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b70e686411943e5b8a4e1ec8753d0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:25 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:42654
I20260810 22:31:25.596776 18223 tablet_replica.cc:330] stopping tablet replica
I20260810 22:31:25.597069 18223 raft_consensus.cc:2229] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:31:25.597306 18223 raft_consensus.cc:2258] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:31:25.598230 18223 ts_tablet_manager.cc:1905] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:31:25.601629 18223 ts_tablet_manager.cc:1918] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:31:25.601730 18223 log.cc:1196] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal/wals/8b70e686411943e5b8a4e1ec8753d0c1
I20260810 22:31:25.602067 18223 ts_tablet_manager.cc:1939] T 8b70e686411943e5b8a4e1ec8753d0c1 P 1125fc90fdb9408c86e5f352ad51f063: Deleting consensus metadata
I20260810 22:31:25.603616 17981 catalog_manager.cc:4841] TS 1125fc90fdb9408c86e5f352ad51f063 (127.1.221.129:33023): tablet 8b70e686411943e5b8a4e1ec8753d0c1 (table test-table [id=264a2e6e644c4e7d9db21e682d65b845]) successfully deleted
W20260810 22:31:25.682741 18218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:25.683048 18218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:25.683125 18218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:25.687197 18218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:25.687294 18218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:25.687388 18218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:31:25.692655 18218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:36617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=34473
--tserver_master_addrs=127.1.221.190:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:25.693923 18218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:25.694998 18218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:25.704547 18232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:25.704589 18230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:25.704560 18229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:25.704836 18218 server_base.cc:1034] running on GCE node
I20260810 22:31:25.705400 18218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:25.706079 18218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:25.707297 18218 hybrid_clock.cc:648] HybridClock initialized: now 1786401085707273 us; error 31 us; skew 500 ppm
Aug 10 22:31:25 dist-test-slave-njxd krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401085, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:25.710327 18218 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:31:25.711215 18218 webserver.cc:460] Webserver started at http://127.1.221.130:34473/ using document root <none> and password file <none>
I20260810 22:31:25.711781 18218 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:25.711854 18218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:25.715660 18218 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:31:25.717670 18242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:25.718920 18218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:31:25.719012 18218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "a996534a529b4a55828ec0ecccc8196e"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "f223810009fba1ab3411e2a7504c6345"
server_key_iv: "518bc18ed38260f73100c15e53733fd2"
server_key_version: "encryptionkey@0"
I20260810 22:31:25.719509 18218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:25.730361 18218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:25.733539 18218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:25.733817 18218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:25.734373 18218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:25.735148 18218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:25.735195 18218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:25.735234 18218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:25.735255 18218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:25.743755 18218 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:36617
I20260810 22:31:25.743791 18365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:36617 every 8 connection(s)
I20260810 22:31:25.744721 18218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:31:25.748629 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 18218
I20260810 22:31:25.749099 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:34951
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:38367
--webserver_port=37031
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:31:25 dist-test-slave-njxd krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401085, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:25.759990 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.746568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:44771 (local address 127.1.221.190:38367)
0810 22:31:25.746745 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:25.746751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:25.747786 (+ 1035us) server_negotiation.cc:407] Connection header received
0810 22:31:25.748666 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:25.748669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:25.748739 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:25.748832 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:25.750218 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.751076 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.752311 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.752597 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.752688 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.755862 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:25.755865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:25.755867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:25.755895 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:25.757530 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:25.758294 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:25.758297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:25.758298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:25.758378 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:25.758803 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:25.758806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:25.758807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:25.758983 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:25.759060 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:25.759749 (+ 689us) server_negotiation.cc:299] Negotiation successful
0810 22:31:25.759865 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 22:31:25.761366 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.746969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:44771)
0810 22:31:25.747632 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:25.747674 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:25.748456 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:25.749025 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:25.749032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:25.749505 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:25.750055 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.750069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.751234 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.751245 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.752148 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.752157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.752780 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.752787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.753025 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.753688 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:25.753717 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:25.755645 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:25.757744 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:25.757755 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:25.757782 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:25.758187 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:25.758478 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:25.758482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:25.758487 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:25.758697 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:25.759096 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:25.759105 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:25.759465 (+ 360us) client_negotiation.cc:769] Sending connection context
0810 22:31:25.759715 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 22:31:25.760130 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":116,"threads_started":1}
I20260810 22:31:25.762758 18366 heartbeater.cc:347] Connected to a master server at 127.1.221.190:38367
I20260810 22:31:25.763018 18366 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:25.763535 18366 heartbeater.cc:511] Master 127.1.221.190:38367 requested a full tablet report, sending...
I20260810 22:31:25.764629 17995 ts_manager.cc:194] Registered new tserver with Master: a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617)
I20260810 22:31:25.765208 17995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:44771
W20260810 22:31:25.866645 18369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:25.866943 18369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:25.867022 18369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:25.871095 18369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:25.871203 18369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:25.871306 18369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:31:25.877632 18369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:34951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=37031
--tserver_master_addrs=127.1.221.190:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:25.878808 18369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:25.879928 18369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:25.889818 18376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:25.889961 18379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:25.889844 18375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:25.889895 18369 server_base.cc:1034] running on GCE node
I20260810 22:31:25.890403 18369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:25.891019 18369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:25.892215 18369 hybrid_clock.cc:648] HybridClock initialized: now 1786401085892196 us; error 28 us; skew 500 ppm
Aug 10 22:31:25 dist-test-slave-njxd krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401085, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:25.895308 18369 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:31:25.896163 18369 webserver.cc:460] Webserver started at http://127.1.221.131:37031/ using document root <none> and password file <none>
I20260810 22:31:25.896719 18369 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:25.896883 18369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:25.900878 18369 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:31:25.903002 18388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:25.904245 18369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:31:25.904374 18369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "00e00b5b89c2439189e3b582b7087bd0"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "83603b69be2c3b5b491fbcdfa4b717c3"
server_key_iv: "fa20ec8c594e4ee5c6d95204bc153de9"
server_key_version: "encryptionkey@0"
I20260810 22:31:25.904896 18369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:25.926038 18369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:25.929644 18369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:25.929833 18369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:25.930537 18369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:25.931430 18369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:25.931481 18369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:25.931553 18369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:25.931591 18369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:25.941073 18369 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:34951
I20260810 22:31:25.941107 18518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:34951 every 8 connection(s)
I20260810 22:31:25.942263 18369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:31:25.947731 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 18369
Aug 10 22:31:25 dist-test-slave-njxd krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401085, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:25.956842 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.948467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35144 (local address 127.1.221.190:38367)
0810 22:31:25.948739 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:25.948744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:25.948889 (+ 145us) server_negotiation.cc:407] Connection header received
0810 22:31:25.949050 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:25.949054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:25.949105 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:25.949203 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:25.950042 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.950564 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.951435 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.951672 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.951798 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.953353 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:25.953356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:25.953361 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:25.953401 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:25.954976 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:25.955493 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:25.955496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:25.955497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:25.955610 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:25.955944 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:25.955946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:25.955948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:25.956216 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:25.956444 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:25.956598 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 22:31:25.956702 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260810 22:31:25.957254 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.944281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:35673 (local address 127.1.221.190:38367)
0810 22:31:25.944426 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:25.944429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:25.945449 (+ 1020us) server_negotiation.cc:407] Connection header received
0810 22:31:25.946451 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:25.946455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:25.946519 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:25.946622 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:25.947908 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.948647 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.949929 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.950163 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.950275 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.953053 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:25.953056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:25.953058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:25.953082 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:25.954860 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:25.955556 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:25.955562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:25.955566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:25.955633 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:25.956058 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:25.956061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:25.956063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:25.956251 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:25.956346 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:25.957005 (+ 659us) server_negotiation.cc:299] Negotiation successful
0810 22:31:25.957109 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48,"spinlock_wait_cycles":768}
I20260810 22:31:25.958384 18522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.944619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:35673)
0810 22:31:25.945289 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:25.945331 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:25.946239 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:25.946815 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:25.946823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:25.947201 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:25.947707 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.947722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.948795 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.948799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.949796 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.949807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.950321 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.950326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.950495 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.951137 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:25.951155 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:25.952861 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:25.954982 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:25.954988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:25.955005 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:25.955462 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:25.955729 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:25.955732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:25.955734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:25.955938 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:25.956353 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:25.956359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:25.956693 (+ 334us) client_negotiation.cc:769] Sending connection context
0810 22:31:25.956925 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 22:31:25.957293 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":118,"threads_started":1}
I20260810 22:31:25.959491 18519 heartbeater.cc:347] Connected to a master server at 127.1.221.190:38367
I20260810 22:31:25.959712 18519 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:25.960165 18519 heartbeater.cc:511] Master 127.1.221.190:38367 requested a full tablet report, sending...
I20260810 22:31:25.961117 17995 ts_manager.cc:194] Registered new tserver with Master: 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951)
I20260810 22:31:25.961710 17995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:35673
I20260810 22:31:25.970986 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:31:25.975358 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.971574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.1.221.190:38367)
0810 22:31:25.971733 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:25.971740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:25.971915 (+ 175us) server_negotiation.cc:407] Connection header received
0810 22:31:25.972055 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:25.972057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:25.972165 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:25.972255 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:31:25.972462 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.973076 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.973963 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.974171 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.974249 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.974502 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:31:25.974919 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:31:25.975012 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:25.975114 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 22:31:25.975166 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 22:31:25.976044 17995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:31:25.979511 17995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:31:25.987392 18138 tablet_service.cc:1467] Processing CreateTablet for tablet b2805a98bb134ac5801f6e38fee75321 (DEFAULT_TABLE table=test-table [id=f3a8c90645ea4e90b4d8144f04d0c5a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:31:25.988080 18138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2805a98bb134ac5801f6e38fee75321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:25.991142 18092 tablet_bootstrap.cc:492] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063: Bootstrap starting.
I20260810 22:31:25.991954 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.986766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36617 (local address 127.0.0.1:60880)
0810 22:31:25.987091 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:25.987104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:25.987201 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:25.988003 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:25.988006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:25.988026 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:25.988119 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.988124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.989411 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.989417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.990347 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.990358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.991484 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.991488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.991583 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.991638 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 22:31:25.991697 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 22:31:25.991795 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260810 22:31:25.992138 18092 tablet_bootstrap.cc:654] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:25.992519 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.987198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60880 (local address 127.1.221.130:36617)
0810 22:31:25.987563 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:25.987577 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:25.987601 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:31:25.987678 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:25.987683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:25.987871 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:25.988033 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:25.988222 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.989249 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.990487 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.991341 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.992139 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.992271 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:25.992329 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 22:31:25.992406 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260810 22:31:25.992940 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.986356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:34951 (local address 127.0.0.1:50360)
0810 22:31:25.986476 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:25.986491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:25.986571 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:25.987570 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:25.987573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:25.987593 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:25.987701 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.987709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.989464 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.989468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.990654 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:25.990666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.992450 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.992455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:25.992598 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.992629 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:31:25.992717 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 22:31:25.992803 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":2}
I20260810 22:31:25.993011 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:25.986564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50360 (local address 127.1.221.131:34951)
0810 22:31:25.986978 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:25.986991 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:25.987019 (+ 28us) server_negotiation.cc:407] Connection header received
0810 22:31:25.987104 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:25.987112 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:25.987358 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:25.987578 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:25.987828 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.989271 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.990796 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:25.991941 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:25.992061 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:25.992229 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:25.992756 (+ 527us) server_negotiation.cc:299] Negotiation successful
0810 22:31:25.992873 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":208,"threads_started":1}
I20260810 22:31:25.994306 18092 tablet_bootstrap.cc:492] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063: No bootstrap required, opened a new log
I20260810 22:31:25.994397 18092 ts_tablet_manager.cc:1397] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:31:25.994866 18092 raft_consensus.cc:348] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:25.994983 18092 raft_consensus.cc:374] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:25.995023 18092 raft_consensus.cc:729] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1125fc90fdb9408c86e5f352ad51f063, State: Initialized, Role: FOLLOWER
I20260810 22:31:25.995150 18092 consensus_queue.cc:260] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:25.995416 18092 ts_tablet_manager.cc:1428] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:31:25.995874 18293 tablet_service.cc:1467] Processing CreateTablet for tablet b2805a98bb134ac5801f6e38fee75321 (DEFAULT_TABLE table=test-table [id=f3a8c90645ea4e90b4d8144f04d0c5a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:31:25.995842 18444 tablet_service.cc:1467] Processing CreateTablet for tablet b2805a98bb134ac5801f6e38fee75321 (DEFAULT_TABLE table=test-table [id=f3a8c90645ea4e90b4d8144f04d0c5a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:31:25.996804 18293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2805a98bb134ac5801f6e38fee75321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:25.996814 18444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2805a98bb134ac5801f6e38fee75321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:26.004354 18533 tablet_bootstrap.cc:492] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0: Bootstrap starting.
I20260810 22:31:26.004462 18535 tablet_bootstrap.cc:492] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e: Bootstrap starting.
I20260810 22:31:26.006996 18533 tablet_bootstrap.cc:654] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:26.007161 18535 tablet_bootstrap.cc:654] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:26.007875 18533 log.cc:824] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:26.008189 18535 log.cc:824] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:26.010219 18533 tablet_bootstrap.cc:492] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0: No bootstrap required, opened a new log
I20260810 22:31:26.010308 18535 tablet_bootstrap.cc:492] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e: No bootstrap required, opened a new log
I20260810 22:31:26.010444 18533 ts_tablet_manager.cc:1397] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:31:26.010453 18535 ts_tablet_manager.cc:1397] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:31:26.014586 18533 raft_consensus.cc:348] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:26.014581 18535 raft_consensus.cc:348] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:26.014906 18533 raft_consensus.cc:374] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:26.014906 18535 raft_consensus.cc:374] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:26.014971 18535 raft_consensus.cc:729] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a996534a529b4a55828ec0ecccc8196e, State: Initialized, Role: FOLLOWER
I20260810 22:31:26.014971 18533 raft_consensus.cc:729] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e00b5b89c2439189e3b582b7087bd0, State: Initialized, Role: FOLLOWER
I20260810 22:31:26.015481 18533 consensus_queue.cc:260] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:26.015481 18535 consensus_queue.cc:260] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:26.016212 18519 heartbeater.cc:503] Master 127.1.221.190:38367 was elected leader, sending a full tablet report...
I20260810 22:31:26.016409 18366 heartbeater.cc:503] Master 127.1.221.190:38367 was elected leader, sending a full tablet report...
I20260810 22:31:26.016590 18533 ts_tablet_manager.cc:1428] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:31:26.016590 18535 ts_tablet_manager.cc:1428] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260810 22:31:26.057444 18215 tablet.cc:2369] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:31:26.193394 18520 tablet.cc:2369] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:31:26.246310 18367 tablet.cc:2369] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:31:26.273666 18538 raft_consensus.cc:482] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:31:26.273923 18538 raft_consensus.cc:504] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:26.275192 18538 leader_election.cc:288] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951), 1125fc90fdb9408c86e5f352ad51f063 (127.1.221.129:33023)
I20260810 22:31:26.279347 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:26.275470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33023 (local address 127.1.221.130:47605)
0810 22:31:26.275814 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:26.275828 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:26.275979 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:26.276350 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:26.276353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:26.276370 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:26.276484 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:26.276488 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:26.277273 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:26.277276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:26.278209 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:26.278219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:26.278893 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:26.278899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:26.278993 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:26.279053 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 22:31:26.279116 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 22:31:26.279170 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":47,"spinlock_wait_cycles":12288,"thread_start_us":79,"threads_started":1}
I20260810 22:31:26.279371 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:26.275614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:47605 (local address 127.1.221.129:33023)
0810 22:31:26.275902 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:26.275907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:26.275923 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:31:26.276132 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:26.276135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:26.276190 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:26.276316 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:26.276623 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:26.277122 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:26.278356 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:26.278788 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:26.278914 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:26.279093 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:26.279136 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 22:31:26.279212 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
I20260810 22:31:26.279547 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:26.275477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:34951 (local address 127.1.221.130:59191)
0810 22:31:26.275816 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:26.275830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:26.275978 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:26.276264 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:26.276278 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:26.276300 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:26.276433 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:26.276440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:26.277212 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:26.277218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:26.278167 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:26.278177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:26.278990 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:26.278994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:26.279171 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:26.279195 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:31:26.279282 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 22:31:26.279347 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":73,"spinlock_wait_cycles":30848,"thread_start_us":88,"threads_started":1}
I20260810 22:31:26.279582 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:26.275572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:59191 (local address 127.1.221.131:34951)
0810 22:31:26.275715 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:26.275720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:26.275927 (+ 207us) server_negotiation.cc:407] Connection header received
0810 22:31:26.276107 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:26.276110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:26.276163 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:26.276222 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:26.276559 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:26.277096 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:26.278293 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:26.278825 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:26.278946 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:26.279004 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:26.279310 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 22:31:26.279430 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:31:26.280330 18163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2805a98bb134ac5801f6e38fee75321" candidate_uuid: "a996534a529b4a55828ec0ecccc8196e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1125fc90fdb9408c86e5f352ad51f063" is_pre_election: true
I20260810 22:31:26.280395 18468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2805a98bb134ac5801f6e38fee75321" candidate_uuid: "a996534a529b4a55828ec0ecccc8196e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e00b5b89c2439189e3b582b7087bd0" is_pre_election: true
I20260810 22:31:26.280714 18163 raft_consensus.cc:2454] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a996534a529b4a55828ec0ecccc8196e in term 0.
I20260810 22:31:26.280730 18468 raft_consensus.cc:2454] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a996534a529b4a55828ec0ecccc8196e in term 0.
I20260810 22:31:26.281255 18243 leader_election.cc:302] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00e00b5b89c2439189e3b582b7087bd0, 1125fc90fdb9408c86e5f352ad51f063, a996534a529b4a55828ec0ecccc8196e; no voters:
I20260810 22:31:26.281560 18538 raft_consensus.cc:2781] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:31:26.281723 18538 raft_consensus.cc:482] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:31:26.281795 18538 raft_consensus.cc:3037] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:26.282863 18538 raft_consensus.cc:504] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:26.283238 18538 leader_election.cc:288] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [CANDIDATE]: Term 1 election: Requested vote from peers 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951), 1125fc90fdb9408c86e5f352ad51f063 (127.1.221.129:33023)
I20260810 22:31:26.283604 18163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2805a98bb134ac5801f6e38fee75321" candidate_uuid: "a996534a529b4a55828ec0ecccc8196e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1125fc90fdb9408c86e5f352ad51f063"
I20260810 22:31:26.283594 18468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2805a98bb134ac5801f6e38fee75321" candidate_uuid: "a996534a529b4a55828ec0ecccc8196e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e00b5b89c2439189e3b582b7087bd0"
I20260810 22:31:26.283758 18163 raft_consensus.cc:3037] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:26.283756 18468 raft_consensus.cc:3037] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:26.284814 18163 raft_consensus.cc:2454] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a996534a529b4a55828ec0ecccc8196e in term 1.
I20260810 22:31:26.284838 18468 raft_consensus.cc:2454] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a996534a529b4a55828ec0ecccc8196e in term 1.
I20260810 22:31:26.285107 18246 leader_election.cc:302] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1125fc90fdb9408c86e5f352ad51f063, a996534a529b4a55828ec0ecccc8196e; no voters:
I20260810 22:31:26.285301 18538 raft_consensus.cc:2781] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:31:26.285570 18538 raft_consensus.cc:686] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [term 1 LEADER]: Becoming Leader. State: Replica: a996534a529b4a55828ec0ecccc8196e, State: Running, Role: LEADER
I20260810 22:31:26.285930 18538 consensus_queue.cc:237] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:26.289022 17995 catalog_manager.cc:5495] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e reported cstate change: term changed from 0 to 1, leader changed from <none> to a996534a529b4a55828ec0ecccc8196e (127.1.221.130). New cstate: current_term: 1 leader_uuid: "a996534a529b4a55828ec0ecccc8196e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } health_report { overall_health: UNKNOWN } } }
I20260810 22:31:26.360952 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:26.357388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.1.221.130:36617)
0810 22:31:26.357553 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:26.357558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:26.357615 (+ 57us) server_negotiation.cc:407] Connection header received
0810 22:31:26.357717 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:26.357720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:26.357779 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:26.357897 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:31:26.358146 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:26.358650 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:26.359500 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:26.359721 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:26.359840 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:26.360151 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:31:26.360560 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:31:26.360700 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:26.360742 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 22:31:26.360801 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 22:31:26.367103 18468 raft_consensus.cc:1261] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0 [term 1 FOLLOWER]: Refusing update from remote peer a996534a529b4a55828ec0ecccc8196e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:31:26.367105 18163 raft_consensus.cc:1261] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Refusing update from remote peer a996534a529b4a55828ec0ecccc8196e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:31:26.367864 18543 consensus_queue.cc:1035] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:31:26.368096 18538 consensus_queue.cc:1035] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:31:26.373703 18548 mvcc.cc:205] Tried to move back new op lower bound from 7317098849751785472 to 7317098849429848064. Current Snapshot: MvccSnapshot[applied={T|T < 7317098849751785472}]
I20260810 22:31:26.374359 18547 mvcc.cc:205] Tried to move back new op lower bound from 7317098849751785472 to 7317098849429848064. Current Snapshot: MvccSnapshot[applied={T|T < 7317098849751785472}]
I20260810 22:31:26.384085 17994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:31:26.384274 17994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:31:26.386724 17994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Sending DeleteTablet for 3 replicas of tablet b2805a98bb134ac5801f6e38fee75321
I20260810 22:31:26.387439 18138 tablet_service.cc:1514] Processing DeleteTablet for tablet b2805a98bb134ac5801f6e38fee75321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:26 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:42654
I20260810 22:31:26.387403 18444 tablet_service.cc:1514] Processing DeleteTablet for tablet b2805a98bb134ac5801f6e38fee75321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:26 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:50360
I20260810 22:31:26.387490 18293 tablet_service.cc:1514] Processing DeleteTablet for tablet b2805a98bb134ac5801f6e38fee75321 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:26 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:60880
I20260810 22:31:26.387971 18559 tablet_replica.cc:330] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063: stopping tablet replica
I20260810 22:31:26.387975 18561 tablet_replica.cc:330] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e: stopping tablet replica
I20260810 22:31:26.388170 18559 raft_consensus.cc:2229] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:31:26.388264 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 18038
I20260810 22:31:26.388288 18561 raft_consensus.cc:2229] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:31:26.388382 18559 raft_consensus.cc:2258] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:31:26.388630 18561 raft_consensus.cc:2258] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:31:26.391851 18561 ts_tablet_manager.cc:1905] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:31:26.392138 18560 tablet_replica.cc:330] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0: stopping tablet replica
I20260810 22:31:26.392452 18560 raft_consensus.cc:2229] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:31:26.392760 18560 raft_consensus.cc:2258] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:31:26.394294 18560 ts_tablet_manager.cc:1905] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:31:26.395738 18561 ts_tablet_manager.cc:1918] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:31:26.395905 18561 log.cc:1196] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/b2805a98bb134ac5801f6e38fee75321
I20260810 22:31:26.396307 18561 ts_tablet_manager.cc:1939] T b2805a98bb134ac5801f6e38fee75321 P a996534a529b4a55828ec0ecccc8196e: Deleting consensus metadata
I20260810 22:31:26.397042 18560 ts_tablet_manager.cc:1918] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:31:26.397195 18560 log.cc:1196] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/b2805a98bb134ac5801f6e38fee75321
I20260810 22:31:26.397604 17980 catalog_manager.cc:4841] TS a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617): tablet b2805a98bb134ac5801f6e38fee75321 (table test-table [id=f3a8c90645ea4e90b4d8144f04d0c5a3]) successfully deleted
I20260810 22:31:26.398029 18560 ts_tablet_manager.cc:1939] T b2805a98bb134ac5801f6e38fee75321 P 00e00b5b89c2439189e3b582b7087bd0: Deleting consensus metadata
I20260810 22:31:26.399273 17977 catalog_manager.cc:4841] TS 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951): tablet b2805a98bb134ac5801f6e38fee75321 (table test-table [id=f3a8c90645ea4e90b4d8144f04d0c5a3]) successfully deleted
W20260810 22:31:26.402041 17981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:33023 (error 108)
I20260810 22:31:26.402114 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 18218
I20260810 22:31:26.409864 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:26.409453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:33023 (local address 127.0.0.1:42664)
0810 22:31:26.409659 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:26.409753 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 22:31:26.410058 17981 catalog_manager.cc:4584] TS 1125fc90fdb9408c86e5f352ad51f063 (127.1.221.129:33023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2805a98bb134ac5801f6e38fee75321: Network error: Client connection negotiation failed: client connection to 127.1.221.129:33023: connect: Connection refused (error 111)
I20260810 22:31:26.414647 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 18369
I20260810 22:31:26.423116 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 17958
I20260810 22:31:26.433780 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:38367
--webserver_interface=127.1.221.190
--webserver_port=45201
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:26.549499 18568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:26.549820 18568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:26.549863 18568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:26.554062 18568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:31:26.554139 18568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:26.554159 18568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:26.554176 18568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:31:26.554193 18568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:31:26.560002 18568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=45201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:26.561197 18568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:26.562458 18568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:26.571283 18573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:26.571280 18576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:26.571322 18574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:26.571360 18568 server_base.cc:1034] running on GCE node
I20260810 22:31:26.572163 18568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:26.573305 18568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:26.574476 18568 hybrid_clock.cc:648] HybridClock initialized: now 1786401086574456 us; error 32 us; skew 500 ppm
Aug 10 22:31:26 dist-test-slave-njxd krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:26.577266 18568 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:31:26.578188 18568 webserver.cc:460] Webserver started at http://127.1.221.190:45201/ using document root <none> and password file <none>
I20260810 22:31:26.578711 18568 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:26.578783 18568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:26.582726 18568 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:31:26.584903 18584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:26.586264 18568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:31:26.586412 18568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "bb6de92ceedb4043a28d7fb9f1a81258"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "35882b7deedcf070c9984a166cedbc61"
server_key_iv: "f394a95d7df8e120d770d0236ca050fd"
server_key_version: "encryptionkey@0"
I20260810 22:31:26.587000 18568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:26.603507 18568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:26.606576 18568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:26.606798 18568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:26.614621 18568 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:38367
I20260810 22:31:26.614918 18639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:38367 every 8 connection(s)
I20260810 22:31:26.615819 18568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:31:26.620291 18640 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:31:26.621426 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 18568
I20260810 22:31:26.621919 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:33023
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:38367
--webserver_port=41339
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:31:26.622164 18640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Bootstrap starting.
I20260810 22:31:26.648370 18640 log.cc:824] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:26.662882 18640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:31:26.663364 18640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258: Bootstrap complete.
I20260810 22:31:26.668186 18640 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } }
I20260810 22:31:26.668620 18640 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6de92ceedb4043a28d7fb9f1a81258, State: Initialized, Role: FOLLOWER
I20260810 22:31:26.669188 18640 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } }
I20260810 22:31:26.669355 18640 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:31:26.669454 18640 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:31:26.669605 18640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 2 FOLLOWER]: Advancing to term 3
I20260810 22:31:26.671597 18640 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } }
I20260810 22:31:26.671947 18640 leader_election.cc:302] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb6de92ceedb4043a28d7fb9f1a81258; no voters:
I20260810 22:31:26.672283 18640 leader_election.cc:288] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 22:31:26.672428 18644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 22:31:26.672694 18644 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [term 3 LEADER]: Becoming Leader. State: Replica: bb6de92ceedb4043a28d7fb9f1a81258, State: Running, Role: LEADER
I20260810 22:31:26.673074 18644 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } }
I20260810 22:31:26.673557 18640 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:31:26.674562 18645 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } } }
I20260810 22:31:26.674711 18645 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:26.675118 18646 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb6de92ceedb4043a28d7fb9f1a81258. Latest consensus state: current_term: 3 leader_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6de92ceedb4043a28d7fb9f1a81258" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 38367 } } }
I20260810 22:31:26.675267 18646 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6de92ceedb4043a28d7fb9f1a81258 [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:26.675617 18654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:31:26.680933 18654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:31:26.681619 18654 catalog_manager.cc:1259] Loaded cluster ID: 5499cecf9fb442ac919ec4f7f82e7b3b
I20260810 22:31:26.681679 18654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 22:31:26.682869 18654 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f868643f9ad google::LogMessage::Fail() at ??:0
@ 0x7f868644288a google::LogMessage::SendToLog() at ??:0
@ 0x7f868643f4de google::LogMessage::Flush() at ??:0
@ 0x7f8686440a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8690d7efdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f8690d7fd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8690d7c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f8690dc0640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f86886913d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8688691c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f86886933f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8688682512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8689cc66db start_thread at ??:0
@ 0x7f8684f3d61f clone at ??:0
W20260810 22:31:26.780378 18642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:26.780656 18642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:26.780699 18642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:26.784621 18642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:26.784672 18642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:26.784727 18642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:31:26.790110 18642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:33023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=41339
--tserver_master_addrs=127.1.221.190:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:26.791152 18642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:26.792188 18642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:26.801990 18693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:26.802047 18690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:26.802284 18691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:26.802579 18642 server_base.cc:1034] running on GCE node
I20260810 22:31:26.803037 18642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:26.803843 18642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:26.805050 18642 hybrid_clock.cc:648] HybridClock initialized: now 1786401086805008 us; error 48 us; skew 500 ppm
Aug 10 22:31:26 dist-test-slave-njxd krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:26.808642 18642 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:31:26.809855 18642 webserver.cc:460] Webserver started at http://127.1.221.129:41339/ using document root <none> and password file <none>
I20260810 22:31:26.810591 18642 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:26.810685 18642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:26.815846 18642 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:31:26.818673 18702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:26.820261 18642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:31:26.820389 18642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "1125fc90fdb9408c86e5f352ad51f063"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "323bebdfbfc808a919a465a02a42a684"
server_key_iv: "dff7bef74debf677d84a89b02e96e3e4"
server_key_version: "encryptionkey@0"
I20260810 22:31:26.821157 18642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:26.872861 18642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:26.876334 18642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:26.876578 18642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:26.877274 18642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:26.878593 18712 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 22:31:26.884454 18642 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 22:31:26.884573 18642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260810 22:31:26.884662 18642 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 22:31:26.887089 18712 tablet_bootstrap.cc:492] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063: Bootstrap starting.
I20260810 22:31:26.886544 18642 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 22:31:26.887454 18642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260810 22:31:26.899106 18642 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:33023
I20260810 22:31:26.899128 18832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:33023 every 8 connection(s)
I20260810 22:31:26.900523 18642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:31:26.902009 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 18642
I20260810 22:31:26.902518 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:36617
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:38367
--webserver_port=34473
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:31:26.904563 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:26.902697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:48867)
0810 22:31:26.903702 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:26.903827 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":135,"threads_started":1}
W20260810 22:31:26.905014 18833 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:38367 (0 consecutive failures): Network error: Failed to ping master at 127.1.221.190:38367: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
I20260810 22:31:26.905776 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:26.905485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:52227)
0810 22:31:26.905650 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:26.905705 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 22:31:26.906502 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:26.906276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:58493)
0810 22:31:26.906386 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:26.906431 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 22:31:26.906766 18833 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 22:31:26.910948 18712 log.cc:824] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:26.921412 18712 tablet_bootstrap.cc:492] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:31:26.921939 18712 tablet_bootstrap.cc:492] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063: Bootstrap complete.
I20260810 22:31:26.923707 18712 ts_tablet_manager.cc:1397] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063: Time spent bootstrapping tablet: real 0.037s user 0.030s sys 0.004s
I20260810 22:31:26.927881 18712 raft_consensus.cc:348] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:26.928467 18712 raft_consensus.cc:729] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1125fc90fdb9408c86e5f352ad51f063, State: Initialized, Role: FOLLOWER
I20260810 22:31:26.929029 18712 consensus_queue.cc:260] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:26.930281 18712 ts_tablet_manager.cc:1428] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 22:31:26.930487 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:26.930186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:36239)
0810 22:31:26.930323 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:26.930393 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 22:31:26.931368 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:26.931080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:34775)
0810 22:31:26.931232 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:26.931294 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 22:31:27.031474 18836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:27.031747 18836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:27.031792 18836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:27.035979 18836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:27.036041 18836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:27.036134 18836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:31:27.041368 18836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:36617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=34473
--tserver_master_addrs=127.1.221.190:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:27.042506 18836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:27.043653 18836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:27.053453 18844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:27.053460 18848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:27.053535 18845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:27.053839 18836 server_base.cc:1034] running on GCE node
I20260810 22:31:27.054435 18836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:27.055126 18836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:27.056316 18836 hybrid_clock.cc:648] HybridClock initialized: now 1786401087056305 us; error 31 us; skew 500 ppm
Aug 10 22:31:27 dist-test-slave-njxd krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:27.059191 18836 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:31:27.060079 18836 webserver.cc:460] Webserver started at http://127.1.221.130:34473/ using document root <none> and password file <none>
I20260810 22:31:27.060655 18836 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:27.060751 18836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:27.064791 18836 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:31:27.067065 18857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:27.068318 18836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:31:27.068460 18836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "a996534a529b4a55828ec0ecccc8196e"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "f223810009fba1ab3411e2a7504c6345"
server_key_iv: "518bc18ed38260f73100c15e53733fd2"
server_key_version: "encryptionkey@0"
I20260810 22:31:27.069072 18836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:27.086349 18836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:27.089465 18836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:27.089705 18836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:27.090338 18836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:27.091178 18836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:27.091250 18836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:27.091326 18836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:27.091379 18836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:27.100803 18836 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:36617
I20260810 22:31:27.100824 18985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:36617 every 8 connection(s)
I20260810 22:31:27.101899 18836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:31:27.104579 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 18836
I20260810 22:31:27.105099 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:34951
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:38367
--webserver_port=37031
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:31:27.106012 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:27.104019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:46311)
0810 22:31:27.104841 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:27.104994 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":120,"threads_started":1}
W20260810 22:31:27.106477 18986 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:38367 (0 consecutive failures): Network error: Failed to ping master at 127.1.221.190:38367: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
I20260810 22:31:27.107136 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:27.106893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:55295)
0810 22:31:27.107015 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:27.107068 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 22:31:27.107882 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:27.107658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:45471)
0810 22:31:27.107767 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:27.107813 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 22:31:27.108093 18986 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:31:27.219844 18990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:27.220090 18990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:27.220134 18990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:27.224227 18990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:27.224293 18990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:27.224350 18990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:31:27.229784 18990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:34951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=37031
--tserver_master_addrs=127.1.221.190:38367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:27.230944 18990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:27.232029 18990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:27.242065 18995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:27.242072 18999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:27.242072 18996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:27.242501 18990 server_base.cc:1034] running on GCE node
I20260810 22:31:27.242937 18990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:27.243537 18990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:27.244699 18990 hybrid_clock.cc:648] HybridClock initialized: now 1786401087244665 us; error 47 us; skew 500 ppm
Aug 10 22:31:27 dist-test-slave-njxd krb5kdc[17333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:27.247527 18990 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:31:27.248407 18990 webserver.cc:460] Webserver started at http://127.1.221.131:37031/ using document root <none> and password file <none>
I20260810 22:31:27.249006 18990 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:27.249104 18990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:27.253022 18990 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:31:27.255199 19006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:27.256572 18990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:31:27.256697 18990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "00e00b5b89c2439189e3b582b7087bd0"
format_stamp: "Formatted at 2026-08-10 22:31:24 on dist-test-slave-njxd"
server_key: "83603b69be2c3b5b491fbcdfa4b717c3"
server_key_iv: "fa20ec8c594e4ee5c6d95204bc153de9"
server_key_version: "encryptionkey@0"
I20260810 22:31:27.257268 18990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:27.272409 18990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:27.276227 18990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:27.276471 18990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:27.277129 18990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:27.278098 18990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:27.278172 18990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:27.278251 18990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:27.278302 18990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:27.287747 18990 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:34951
I20260810 22:31:27.287825 19132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:34951 every 8 connection(s)
I20260810 22:31:27.288817 18990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:31:27.292469 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:27.291005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:50873)
0810 22:31:27.291668 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:27.291784 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":116,"threads_started":1}
W20260810 22:31:27.292987 19134 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:38367 (0 consecutive failures): Network error: Failed to ping master at 127.1.221.190:38367: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
I20260810 22:31:27.293507 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:27.293307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:33051)
0810 22:31:27.293412 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:27.293455 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 22:31:27.293705 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 18990
I20260810 22:31:27.294163 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:27.293940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:59615)
0810 22:31:27.294051 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:27.294112 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 22:31:27.294395 19134 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 22:31:27.932592 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:27.932168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:49107)
0810 22:31:27.932417 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:27.932497 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260810 22:31:28.109179 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:28.108765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:46871)
0810 22:31:28.109013 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:28.109071 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260810 22:31:28.214958 19140 raft_consensus.cc:482] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:31:28.215205 19140 raft_consensus.cc:504] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:28.216377 19140 leader_election.cc:288] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951), a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617)
Aug 10 22:31:28 dist-test-slave-njxd krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
Aug 10 22:31:28 dist-test-slave-njxd krb5kdc[17333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
I20260810 22:31:28.233357 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:28.216803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:60737 (local address 127.1.221.131:34951)
0810 22:31:28.217549 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:28.217562 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:28.217616 (+ 54us) server_negotiation.cc:407] Connection header received
0810 22:31:28.218434 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:28.218444 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:28.218895 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:28.219232 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:28.221267 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:28.222884 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:28.224842 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:28.225512 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:28.225726 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:28.228591 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:28.228595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:28.228608 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:28.228635 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:28.230809 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:28.231310 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:28.231315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:28.231319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:28.231432 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:28.231803 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:28.231807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:28.231809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:28.232200 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:28.232350 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:28.232890 (+ 540us) server_negotiation.cc:299] Negotiation successful
0810 22:31:28.233136 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":121,"threads_started":1}
I20260810 22:31:28.233373 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:28.216555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:34951 (local address 127.1.221.129:60737)
0810 22:31:28.216750 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:28.216793 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:28.217315 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:28.219334 (+ 2019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:28.219344 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:28.220085 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:28.221031 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:28.221054 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:28.223060 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:28.223064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:28.224666 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:28.224679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:28.225677 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:28.225682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:28.225900 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:28.226561 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:28.226578 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:28.228350 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:28.230940 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:28.230945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:28.230956 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:28.231201 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:28.231547 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:28.231550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:28.231552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:28.231693 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:28.232336 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:28.232342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:28.232573 (+ 231us) client_negotiation.cc:769] Sending connection context
0810 22:31:28.232778 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 22:31:28.233131 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":412,"spinlock_wait_cycles":52224}
I20260810 22:31:28.233388 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:28.216868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:52659 (local address 127.1.221.130:36617)
0810 22:31:28.217279 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:28.217303 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:28.217345 (+ 42us) server_negotiation.cc:407] Connection header received
0810 22:31:28.218440 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:28.218446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:28.218895 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:28.219232 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:28.221266 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:28.222884 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:28.224816 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:28.225467 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:28.225664 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:28.228885 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:28.228892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:28.228908 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:28.228948 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:28.230809 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:28.231561 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:28.231567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:28.231572 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:28.231691 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:28.231996 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:28.232000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:28.232002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:28.232395 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:28.232539 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:28.232905 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 22:31:28.233183 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":63,"threads_started":1}
I20260810 22:31:28.233404 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:28.216636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36617 (local address 127.1.221.129:52659)
0810 22:31:28.217026 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:28.217039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:28.217312 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:28.219334 (+ 2022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:28.219344 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:28.220085 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:28.221031 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:28.221054 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:28.223060 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:28.223064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:28.224666 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:28.224679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:28.225657 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:28.225661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:28.225905 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:28.226624 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:28.226637 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:28.228668 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:28.231292 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:28.231294 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:28.231295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:28.231448 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:28.231818 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:28.231821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:28.231822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:28.231896 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:28.232523 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:28.232526 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:28.232626 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 22:31:28.232771 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 22:31:28.233218 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":404,"spinlock_wait_cycles":3072,"thread_start_us":74,"threads_started":1}
W20260810 22:31:28.236207 18707 leader_election.cc:341] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617): Not found: Tablet not found: b2805a98bb134ac5801f6e38fee75321
W20260810 22:31:28.236459 18704 leader_election.cc:341] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951): Not found: Tablet not found: b2805a98bb134ac5801f6e38fee75321
I20260810 22:31:28.236629 18704 leader_election.cc:302] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1125fc90fdb9408c86e5f352ad51f063; no voters: 00e00b5b89c2439189e3b582b7087bd0, a996534a529b4a55828ec0ecccc8196e
I20260810 22:31:28.236943 19140 raft_consensus.cc:2726] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:31:28.295293 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:28.294908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:57787)
0810 22:31:28.295142 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:28.295192 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
I20260810 22:31:28.934026 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:28.933525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:55129)
0810 22:31:28.933859 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:28.933925 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":144,"threads_started":1}
I20260810 22:31:29.110428 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:29.109979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:54835)
0810 22:31:29.110269 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:29.110331 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
I20260810 22:31:29.296618 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:29.296083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:39703)
0810 22:31:29.296437 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:29.296509 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":77,"threads_started":1}
I20260810 22:31:29.935418 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:29.934900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:39777)
0810 22:31:29.935243 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:29.935319 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260810 22:31:30.111727 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:30.111317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:33931)
0810 22:31:30.111560 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:30.111615 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260810 22:31:30.132985 19150 raft_consensus.cc:482] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:31:30.133122 19150 raft_consensus.cc:504] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:30.133621 19150 leader_election.cc:288] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951), a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617)
W20260810 22:31:30.134474 18707 leader_election.cc:341] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617): Not found: Tablet not found: b2805a98bb134ac5801f6e38fee75321
W20260810 22:31:30.134615 18704 leader_election.cc:341] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951): Not found: Tablet not found: b2805a98bb134ac5801f6e38fee75321
I20260810 22:31:30.134742 18704 leader_election.cc:302] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1125fc90fdb9408c86e5f352ad51f063; no voters: 00e00b5b89c2439189e3b582b7087bd0, a996534a529b4a55828ec0ecccc8196e
I20260810 22:31:30.134915 19150 raft_consensus.cc:2726] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:31:30.297856 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:30.297358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:40851)
0810 22:31:30.297689 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:30.297757 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260810 22:31:30.936575 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:30.936130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:36843)
0810 22:31:30.936422 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:30.936479 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
I20260810 22:31:31.113004 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:31.112479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:51099)
0810 22:31:31.112851 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:31.112910 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":77,"threads_started":1}
I20260810 22:31:31.299149 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:31.298718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:49395)
0810 22:31:31.298989 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:31.299060 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":96,"threads_started":1}
I20260810 22:31:31.821774 19155 raft_consensus.cc:482] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:31:31.821931 19155 raft_consensus.cc:504] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:31.822343 19155 leader_election.cc:288] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951), a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617)
W20260810 22:31:31.823186 18707 leader_election.cc:341] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617): Not found: Tablet not found: b2805a98bb134ac5801f6e38fee75321
W20260810 22:31:31.823297 18704 leader_election.cc:341] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951): Not found: Tablet not found: b2805a98bb134ac5801f6e38fee75321
I20260810 22:31:31.823361 18704 leader_election.cc:302] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1125fc90fdb9408c86e5f352ad51f063; no voters: 00e00b5b89c2439189e3b582b7087bd0, a996534a529b4a55828ec0ecccc8196e
I20260810 22:31:31.823482 19155 raft_consensus.cc:2726] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:31:31.938115 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:31.937650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:34731)
0810 22:31:31.937948 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:31.938013 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
I20260810 22:31:32.114123 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:32.113739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:40591)
0810 22:31:32.113985 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:32.114039 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260810 22:31:32.300464 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:32.300024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:57881)
0810 22:31:32.300309 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:32.300368 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":99,"threads_started":1}
I20260810 22:31:32.939368 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:32.938888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:37995)
0810 22:31:32.939195 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:32.939264 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260810 22:31:33.115393 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:33.114917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:54155)
0810 22:31:33.115237 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:33.115298 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":134,"threads_started":1}
I20260810 22:31:33.301632 19161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:33.301186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:57705)
0810 22:31:33.301432 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:33.301487 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260810 22:31:33.940678 19162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:33.940269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:57047)
0810 22:31:33.940519 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:33.940579 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20260810 22:31:34.116641 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:34.116166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:49685)
0810 22:31:34.116497 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:34.116551 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
I20260810 22:31:34.302902 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:34.302427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:57375)
0810 22:31:34.302744 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:34.302804 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":78,"threads_started":1}
I20260810 22:31:34.942011 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:34.941557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:41005)
0810 22:31:34.941851 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:34.941913 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":76,"threads_started":1}
I20260810 22:31:35.117935 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:35.117446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:48539)
0810 22:31:35.117773 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:35.117835 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":161,"threads_started":1}
I20260810 22:31:35.304350 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:35.303868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:54095)
0810 22:31:35.304165 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:35.304234 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
I20260810 22:31:35.943287 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:35.942853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:44939)
0810 22:31:35.943117 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:35.943183 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260810 22:31:36.119237 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:36.118776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:47733)
0810 22:31:36.119051 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:36.119127 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
I20260810 22:31:36.208010 19173 raft_consensus.cc:482] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:31:36.208185 19173 raft_consensus.cc:504] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:36.208693 19173 leader_election.cc:288] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951), a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617)
W20260810 22:31:36.209510 18707 leader_election.cc:341] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617): Not found: Tablet not found: b2805a98bb134ac5801f6e38fee75321
W20260810 22:31:36.209637 18704 leader_election.cc:341] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951): Not found: Tablet not found: b2805a98bb134ac5801f6e38fee75321
I20260810 22:31:36.209729 18704 leader_election.cc:302] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1125fc90fdb9408c86e5f352ad51f063; no voters: 00e00b5b89c2439189e3b582b7087bd0, a996534a529b4a55828ec0ecccc8196e
I20260810 22:31:36.209892 19173 raft_consensus.cc:2726] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:31:36.305864 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:36.305372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:49757)
0810 22:31:36.305701 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:36.305766 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":86,"threads_started":1}
I20260810 22:31:36.944715 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:36.944252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:49839)
0810 22:31:36.944542 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:36.944608 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
I20260810 22:31:37.120558 19176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:37.120111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:57817)
0810 22:31:37.120400 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:37.120460 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":100,"threads_started":1}
I20260810 22:31:37.307053 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:37.306643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:49057)
0810 22:31:37.306896 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:37.306956 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":81,"threads_started":1}
I20260810 22:31:37.945956 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:37.945487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:39207)
0810 22:31:37.945795 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:37.945856 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":130,"threads_started":1}
I20260810 22:31:38.103881 19179 raft_consensus.cc:482] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:31:38.104060 19179 raft_consensus.cc:504] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e00b5b89c2439189e3b582b7087bd0" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34951 } } peers { permanent_uuid: "a996534a529b4a55828ec0ecccc8196e" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36617 } } peers { permanent_uuid: "1125fc90fdb9408c86e5f352ad51f063" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 33023 } }
I20260810 22:31:38.104529 19179 leader_election.cc:288] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951), a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617)
W20260810 22:31:38.105324 18707 leader_election.cc:341] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a996534a529b4a55828ec0ecccc8196e (127.1.221.130:36617): Not found: Tablet not found: b2805a98bb134ac5801f6e38fee75321
W20260810 22:31:38.105430 18704 leader_election.cc:341] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00e00b5b89c2439189e3b582b7087bd0 (127.1.221.131:34951): Not found: Tablet not found: b2805a98bb134ac5801f6e38fee75321
I20260810 22:31:38.105489 18704 leader_election.cc:302] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1125fc90fdb9408c86e5f352ad51f063; no voters: 00e00b5b89c2439189e3b582b7087bd0, a996534a529b4a55828ec0ecccc8196e
I20260810 22:31:38.105677 19179 raft_consensus.cc:2726] T b2805a98bb134ac5801f6e38fee75321 P 1125fc90fdb9408c86e5f352ad51f063 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:31:38.121896 19180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:38.121301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:48871)
0810 22:31:38.121564 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:38.121786 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20260810 22:31:38.308449 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:38.308038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:45055)
0810 22:31:38.308293 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:38.308348 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260810 22:31:38.947207 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:38.946736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:57553)
0810 22:31:38.947034 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:38.947102 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":83,"threads_started":1}
I20260810 22:31:39.123169 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:39.122724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:33005)
0810 22:31:39.123004 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:39.123070 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
I20260810 22:31:39.309818 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:39.309323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:38097)
0810 22:31:39.309652 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:39.309717 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260810 22:31:39.948508 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:39.947959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:47099)
0810 22:31:39.948340 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:39.948406 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":100,"threads_started":1}
I20260810 22:31:40.124399 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:40.123960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:56943)
0810 22:31:40.124240 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:40.124303 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":105,"threads_started":1}
I20260810 22:31:40.311120 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:40.310662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:34635)
0810 22:31:40.310957 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:40.311020 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
I20260810 22:31:40.949848 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:40.949364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:43671)
0810 22:31:40.949685 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:40.949749 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260810 22:31:41.125877 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:41.125364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:46983)
0810 22:31:41.125684 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:41.125748 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
I20260810 22:31:41.312386 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:41.311935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:43051)
0810 22:31:41.312232 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:41.312288 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
I20260810 22:31:41.951243 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:41.950785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.129:53653)
0810 22:31:41.951078 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:41.951148 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":74,"threads_started":1}
I20260810 22:31:42.127226 19193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:42.126715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.130:34637)
0810 22:31:42.127028 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:42.127119 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260810 22:31:42.303714 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 18642
I20260810 22:31:42.313694 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 18836
I20260810 22:31:42.313990 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:42.313470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:38367 (local address 127.1.221.131:52245)
0810 22:31:42.313768 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:42.313859 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":122,"threads_started":1}
I20260810 22:31:42.320782 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 18990
2026-08-10T22:31:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19536 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:31:42 dist-test-slave-njxd krb5kdc[19197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:31:42 dist-test-slave-njxd krb5kdc[19197](info): set up 2 sockets
Aug 10 22:31:42 dist-test-slave-njxd krb5kdc[19197](info): commencing operation
krb5kdc: starting...
W20260810 22:31:43.414352 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:31:43 dist-test-slave-njxd krb5kdc[19197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:31:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:31:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:31:43.542383 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44603
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:44603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:43.655843 19213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:43.656198 19213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:43.656286 19213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:43.660403 19213 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 22:31:43.660491 19213 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 22:31:43.660518 19213 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 22:31:43.660542 19213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:31:43.660584 19213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:43.660630 19213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:43.660665 19213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:31:43.660701 19213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:31:43.666137 19213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:44603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:43.667325 19213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:43.668457 19213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:43.677551 19222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:43.677620 19218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:43.677616 19213 server_base.cc:1034] running on GCE node
W20260810 22:31:43.677774 19220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:43.678503 19213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:43.679621 19213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:43.680805 19213 hybrid_clock.cc:648] HybridClock initialized: now 1786401103680793 us; error 24 us; skew 500 ppm
Aug 10 22:31:43 dist-test-slave-njxd krb5kdc[19197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401103, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:43.683570 19213 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:31:43.684525 19213 webserver.cc:460] Webserver started at http://127.1.221.190:44949/ using document root <none> and password file <none>
I20260810 22:31:43.685086 19213 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:43.685174 19213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:43.685411 19213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:31:43.687525 19213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "4d87813547aa4c75bc18d27acccaacf4"
format_stamp: "Formatted at 2026-08-10 22:31:43 on dist-test-slave-njxd"
server_key: "bdd61356327bcbeb314f278bc31abac2"
server_key_iv: "167df6b76ce225735ed0c403cc6595a2"
server_key_version: "encryptionkey@0"
I20260810 22:31:43.688098 19213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "4d87813547aa4c75bc18d27acccaacf4"
format_stamp: "Formatted at 2026-08-10 22:31:43 on dist-test-slave-njxd"
server_key: "bdd61356327bcbeb314f278bc31abac2"
server_key_iv: "167df6b76ce225735ed0c403cc6595a2"
server_key_version: "encryptionkey@0"
I20260810 22:31:43.691972 19213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260810 22:31:43.694355 19229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:43.695550 19213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:31:43.695689 19213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "4d87813547aa4c75bc18d27acccaacf4"
format_stamp: "Formatted at 2026-08-10 22:31:43 on dist-test-slave-njxd"
server_key: "bdd61356327bcbeb314f278bc31abac2"
server_key_iv: "167df6b76ce225735ed0c403cc6595a2"
server_key_version: "encryptionkey@0"
I20260810 22:31:43.695797 19213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:43.722074 19213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:43.724975 19213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:43.725190 19213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:43.732857 19213 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:44603
I20260810 22:31:43.733158 19290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:44603 every 8 connection(s)
I20260810 22:31:43.733898 19213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:31:43.736920 19292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:43.739707 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 19213
I20260810 22:31:43.739852 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:31:43.740145 1910 external_mini_cluster.cc:1428] Setting key 97fc397c1851e1c11b650da1e93090e8
I20260810 22:31:43.743079 19292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Bootstrap starting.
I20260810 22:31:43.745865 19292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:43.746780 19292 log.cc:824] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:43.748832 19292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: No bootstrap required, opened a new log
Aug 10 22:31:43 dist-test-slave-njxd krb5kdc[19197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:43.752890 19292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } }
I20260810 22:31:43.753125 19292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:43.753160 19292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d87813547aa4c75bc18d27acccaacf4, State: Initialized, Role: FOLLOWER
I20260810 22:31:43.753734 19292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } }
I20260810 22:31:43.753922 19292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:31:43.754034 19292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:31:43.754158 19292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:43.754976 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:43.741555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.1.221.190:44603)
0810 22:31:43.742050 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:43.742067 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:43.742104 (+ 37us) server_negotiation.cc:407] Connection header received
0810 22:31:43.742746 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:43.742763 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:43.743114 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:43.743439 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:43.744371 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:43.745380 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:43.746618 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:43.747158 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:43.747335 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:43.750120 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:43.750125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:43.750140 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:43.750167 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:43.751927 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:43.752438 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:43.752443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:43.752448 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:43.752555 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:43.752856 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:43.752858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:43.752859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:43.753343 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:43.753560 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:43.753812 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 22:31:43.754139 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":139,"threads_started":1}
I20260810 22:31:43.755206 19292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } }
I20260810 22:31:43.755582 19292 leader_election.cc:302] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d87813547aa4c75bc18d27acccaacf4; no voters:
I20260810 22:31:43.755975 19292 leader_election.cc:288] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:31:43.756026 19297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:31:43.756255 19297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 1 LEADER]: Becoming Leader. State: Replica: 4d87813547aa4c75bc18d27acccaacf4, State: Running, Role: LEADER
I20260810 22:31:43.756623 19297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } }
I20260810 22:31:43.757333 19292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:31:43.757903 19297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d87813547aa4c75bc18d27acccaacf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } } }
I20260810 22:31:43.758055 19297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:43.758466 19298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d87813547aa4c75bc18d27acccaacf4. Latest consensus state: current_term: 1 leader_uuid: "4d87813547aa4c75bc18d27acccaacf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } } }
I20260810 22:31:43.758580 19298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:31:43.758966 19303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:31:43.762117 19303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:31:43.767944 19303 catalog_manager.cc:1347] Generated new cluster ID: a0357ccd43654cf294ad946082679dc0
I20260810 22:31:43.768013 19303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:31:43.789536 19303 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:31:43.790563 19303 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:31:43.798681 19303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Generated new TSK 0
I20260810 22:31:43.799460 19303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:31:43.839907 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:44603
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:43.975528 19320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:43.975838 19320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:43.975910 19320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:43.980072 19320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:43.980158 19320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:43.980247 19320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:31:43.986130 19320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:44603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:43.987283 19320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:43.988344 19320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:43.998073 19326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:43.998080 19327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:43.998088 19329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:43.998133 19320 server_base.cc:1034] running on GCE node
I20260810 22:31:43.998788 19320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:43.999413 19320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:44.000582 19320 hybrid_clock.cc:648] HybridClock initialized: now 1786401104000569 us; error 34 us; skew 500 ppm
Aug 10 22:31:44 dist-test-slave-njxd krb5kdc[19197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401104, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:44.003536 19320 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:31:44.004344 19320 webserver.cc:460] Webserver started at http://127.1.221.129:33749/ using document root <none> and password file <none>
I20260810 22:31:44.004873 19320 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:44.004930 19320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:44.005092 19320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:31:44.007107 19320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "9ef20c5cdc4342d895474b7513b901e4"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "33af3b9be5af579681d6c77d0cc654ca"
server_key_iv: "58582788f6f70bdebb67d12569932197"
server_key_version: "encryptionkey@0"
I20260810 22:31:44.007699 19320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "9ef20c5cdc4342d895474b7513b901e4"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "33af3b9be5af579681d6c77d0cc654ca"
server_key_iv: "58582788f6f70bdebb67d12569932197"
server_key_version: "encryptionkey@0"
I20260810 22:31:44.011492 19320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 22:31:44.013923 19337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:44.015183 19320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:31:44.015303 19320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "9ef20c5cdc4342d895474b7513b901e4"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "33af3b9be5af579681d6c77d0cc654ca"
server_key_iv: "58582788f6f70bdebb67d12569932197"
server_key_version: "encryptionkey@0"
I20260810 22:31:44.015407 19320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:44.033061 19320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:44.036533 19320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:44.036742 19320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:44.037374 19320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:44.038311 19320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:44.038359 19320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:44.038424 19320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:44.038456 19320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:44.047439 19320 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:40933
I20260810 22:31:44.047472 19468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:40933 every 8 connection(s)
I20260810 22:31:44.048398 19320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
Aug 10 22:31:44 dist-test-slave-njxd krb5kdc[19197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401104, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:44.058439 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 19320
I20260810 22:31:44.058610 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:31:44.058919 1910 external_mini_cluster.cc:1428] Setting key 198511b1cf857dbcabfced5726ec7ee0
I20260810 22:31:44.062337 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.050197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:42227 (local address 127.1.221.190:44603)
0810 22:31:44.050349 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:44.050354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:44.051190 (+ 836us) server_negotiation.cc:407] Connection header received
0810 22:31:44.052554 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:44.052558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:44.052618 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:44.052717 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:44.053975 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.054526 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.055683 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.055918 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.056064 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.058733 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:44.058736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:44.058738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:44.058761 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:44.060286 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:44.060799 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:44.060809 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:44.060810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:44.060887 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:44.061214 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:44.061218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:44.061220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:44.061394 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:44.061500 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:44.062016 (+ 516us) server_negotiation.cc:299] Negotiation successful
0810 22:31:44.062167 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:31:44.063102 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.050563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:42227)
0810 22:31:44.051028 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:44.051071 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:44.052331 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:44.052893 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:44.052900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:44.053289 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:44.053830 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.053844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.054662 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.054666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.055554 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.055563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.056108 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.056113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.056282 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.056818 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:44.056834 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:44.058473 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:44.060407 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:44.060413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:44.060428 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:44.060694 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:44.060976 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:44.060979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:44.060981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:44.061123 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:44.061498 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:44.061505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:44.061792 (+ 287us) client_negotiation.cc:769] Sending connection context
0810 22:31:44.061976 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 22:31:44.062301 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":115,"threads_started":1}
I20260810 22:31:44.064396 19469 heartbeater.cc:347] Connected to a master server at 127.1.221.190:44603
I20260810 22:31:44.064651 19469 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:44.065230 19469 heartbeater.cc:511] Master 127.1.221.190:44603 requested a full tablet report, sending...
I20260810 22:31:44.067034 19249 ts_manager.cc:194] Registered new tserver with Master: 9ef20c5cdc4342d895474b7513b901e4 (127.1.221.129:40933)
I20260810 22:31:44.068645 19249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:42227
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:31:44.087898 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:44603
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:44.199208 19474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:44.199427 19474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:44.199468 19474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:44.203413 19474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:44.203459 19474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:44.203553 19474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:31:44.208298 19474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:44603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:44.209260 19474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:44.210315 19474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:31:44.219527 19474 server_base.cc:1034] running on GCE node
W20260810 22:31:44.219511 19481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:44.219503 19479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:44.219537 19483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:44.220254 19474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:44.220875 19474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:44.222046 19474 hybrid_clock.cc:648] HybridClock initialized: now 1786401104222031 us; error 26 us; skew 500 ppm
Aug 10 22:31:44 dist-test-slave-njxd krb5kdc[19197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401104, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:44.224666 19474 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:31:44.225524 19474 webserver.cc:460] Webserver started at http://127.1.221.130:40529/ using document root <none> and password file <none>
I20260810 22:31:44.226078 19474 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:44.226176 19474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:44.226394 19474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:31:44.228369 19474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "9b0673f9c4c14dc9b244fda13157a1dc"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "8a36b533fd3a1367ffa681a5bdd26bda"
server_key_iv: "be37f9e940d5057fe9fdc325c196435b"
server_key_version: "encryptionkey@0"
I20260810 22:31:44.228924 19474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "9b0673f9c4c14dc9b244fda13157a1dc"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "8a36b533fd3a1367ffa681a5bdd26bda"
server_key_iv: "be37f9e940d5057fe9fdc325c196435b"
server_key_version: "encryptionkey@0"
I20260810 22:31:44.232656 19474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 22:31:44.235009 19491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:44.236155 19474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:31:44.236284 19474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "9b0673f9c4c14dc9b244fda13157a1dc"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "8a36b533fd3a1367ffa681a5bdd26bda"
server_key_iv: "be37f9e940d5057fe9fdc325c196435b"
server_key_version: "encryptionkey@0"
I20260810 22:31:44.236385 19474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:44.251421 19474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:44.255208 19474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:44.255409 19474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:44.255997 19474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:44.256811 19474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:44.256894 19474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:44.256963 19474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:44.257011 19474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:44.266463 19474 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:36143
I20260810 22:31:44.266489 19620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:36143 every 8 connection(s)
I20260810 22:31:44.267352 19474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:31:44.275077 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 19474
I20260810 22:31:44.275218 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:31:44.275498 1910 external_mini_cluster.cc:1428] Setting key a01c9f19d710394dd58cab8f97f841f0
Aug 10 22:31:44 dist-test-slave-njxd krb5kdc[19197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401104, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:44.281786 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.269110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:34715 (local address 127.1.221.190:44603)
0810 22:31:44.269253 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:44.269257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:44.270087 (+ 830us) server_negotiation.cc:407] Connection header received
0810 22:31:44.271423 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:44.271427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:44.271485 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:44.271590 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:44.272783 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.273326 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.274609 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.274846 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.274960 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.277767 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:44.277773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:44.277779 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:44.277812 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:44.279466 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:44.280090 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:44.280092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:44.280094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:44.280170 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:44.280583 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:44.280585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:44.280586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:44.280776 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:44.280857 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:44.281501 (+ 644us) server_negotiation.cc:299] Negotiation successful
0810 22:31:44.281666 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:31:44.283007 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.269463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:34715)
0810 22:31:44.269939 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:44.269985 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:44.271212 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:44.271767 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:44.271774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:44.272140 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:44.272635 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.272648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.273455 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.273459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.274453 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.274462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.275000 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.275004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.275197 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.275858 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:44.275874 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:44.277553 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:44.279582 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:44.279588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:44.279605 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:44.279983 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:44.280256 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:44.280261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:44.280265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:44.280478 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:44.280886 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:44.280896 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:44.281217 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 22:31:44.281433 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 22:31:44.281817 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260810 22:31:44.284386 19621 heartbeater.cc:347] Connected to a master server at 127.1.221.190:44603
I20260810 22:31:44.284689 19621 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:44.285187 19621 heartbeater.cc:511] Master 127.1.221.190:44603 requested a full tablet report, sending...
I20260810 22:31:44.286314 19249 ts_manager.cc:194] Registered new tserver with Master: 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130:36143)
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260810 22:31:44.286962 19249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:34715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:31:44.304881 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:44603
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:31:44.417995 19626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:31:44.418239 19626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:31:44.418285 19626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:31:44.422537 19626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:31:44.422607 19626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:31:44.422693 19626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:31:44.428105 19626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:44603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:31:44.429198 19626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:31:44.430378 19626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:31:44.440272 19631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:44.440443 19626 server_base.cc:1034] running on GCE node
W20260810 22:31:44.440297 19635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:31:44.440297 19632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:31:44.441017 19626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:31:44.441715 19626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:31:44.442902 19626 hybrid_clock.cc:648] HybridClock initialized: now 1786401104442875 us; error 49 us; skew 500 ppm
Aug 10 22:31:44 dist-test-slave-njxd krb5kdc[19197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401104, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:31:44.445854 19626 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:31:44.446856 19626 webserver.cc:460] Webserver started at http://127.1.221.131:43219/ using document root <none> and password file <none>
I20260810 22:31:44.447436 19626 fs_manager.cc:359] Metadata directory not provided
I20260810 22:31:44.447526 19626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:31:44.447746 19626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:31:44.449949 19626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "86aae5efd275476fa0cd52dd3b0edb9d"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "6edb581c84455be3b82c3660f767151f"
server_key_iv: "7f992e4d6d55f87c7faacb34dcdde0d8"
server_key_version: "encryptionkey@0"
I20260810 22:31:44.450564 19626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "86aae5efd275476fa0cd52dd3b0edb9d"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "6edb581c84455be3b82c3660f767151f"
server_key_iv: "7f992e4d6d55f87c7faacb34dcdde0d8"
server_key_version: "encryptionkey@0"
I20260810 22:31:44.454488 19626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:31:44.456939 19642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:44.458333 19626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:31:44.458461 19626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "86aae5efd275476fa0cd52dd3b0edb9d"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "6edb581c84455be3b82c3660f767151f"
server_key_iv: "7f992e4d6d55f87c7faacb34dcdde0d8"
server_key_version: "encryptionkey@0"
I20260810 22:31:44.458585 19626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:31:44.472855 19626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:31:44.476313 19626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:31:44.476523 19626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:31:44.477159 19626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:31:44.478134 19626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:31:44.478201 19626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:44.478274 19626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:31:44.478323 19626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:31:44.487488 19626 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:43589
I20260810 22:31:44.487526 19766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:43589 every 8 connection(s)
I20260810 22:31:44.488409 19626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:31:44.491429 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 19626
I20260810 22:31:44.491562 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:31:44.491922 1910 external_mini_cluster.cc:1428] Setting key 44f17236ae6f71c992061c4add4d3f35
Aug 10 22:31:44 dist-test-slave-njxd krb5kdc[19197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401104, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:31:44.502238 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.490284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:60947 (local address 127.1.221.190:44603)
0810 22:31:44.490424 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:44.490428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:44.491793 (+ 1365us) server_negotiation.cc:407] Connection header received
0810 22:31:44.492911 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:44.492916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:44.492978 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:44.493048 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:44.494513 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.495055 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.496121 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.496331 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.496405 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.498783 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:44.498786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:44.498788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:44.498811 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:44.500429 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:44.500882 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:44.500884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:44.500886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:44.500959 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:44.501242 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:44.501244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:44.501245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:44.501419 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:44.501494 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:44.502018 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 22:31:44.502118 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:31:44.503098 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.490643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:60947)
0810 22:31:44.491613 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:44.491674 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:44.492683 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:44.493275 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:44.493285 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:44.493775 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:31:44.494388 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.494402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.495160 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.495163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.496019 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.496027 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.496454 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.496457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.496596 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.497137 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:31:44.497153 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:31:44.498643 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:31:44.500531 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:44.500536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:44.500547 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:44.500797 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:44.501033 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:31:44.501035 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:31:44.501036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:31:44.501169 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:31:44.501501 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:31:44.501505 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:31:44.501808 (+ 303us) client_negotiation.cc:769] Sending connection context
0810 22:31:44.501968 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 22:31:44.502221 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":565,"threads_started":1}
I20260810 22:31:44.504213 19767 heartbeater.cc:347] Connected to a master server at 127.1.221.190:44603
I20260810 22:31:44.504460 19767 heartbeater.cc:464] Registering TS with master...
I20260810 22:31:44.504992 19767 heartbeater.cc:511] Master 127.1.221.190:44603 requested a full tablet report, sending...
I20260810 22:31:44.506008 19249 ts_manager.cc:194] Registered new tserver with Master: 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589)
I20260810 22:31:44.506597 19249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:60947
I20260810 22:31:44.517949 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:31:44.527019 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.520048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.1.221.190:44603)
0810 22:31:44.520168 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:44.520171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:44.520334 (+ 163us) server_negotiation.cc:407] Connection header received
0810 22:31:44.520424 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:44.520426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:44.520465 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:44.520534 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:31:44.521215 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.521774 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.522540 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.522741 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.522818 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.524091 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:31:44.524094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:31:44.524096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:31:44.524118 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:31:44.525558 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:44.525975 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:44.525977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:44.525978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:44.526055 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:31:44.526300 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:31:44.526302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:31:44.526303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:31:44.526509 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:31:44.526610 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:44.526811 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 22:31:44.526904 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:31:44.529846 19249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:31:44.532897 19249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:31:44.547325 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.542247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.1.221.129:40933)
0810 22:31:44.542614 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:44.542633 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:44.542842 (+ 209us) server_negotiation.cc:407] Connection header received
0810 22:31:44.543067 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:44.543075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:44.543287 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:44.543441 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:44.543622 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.544602 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.546144 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.546675 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.546793 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.546934 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:44.547086 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 22:31:44.547180 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":75,"threads_started":1}
I20260810 22:31:44.547314 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.542065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:40933 (local address 127.0.0.1:43028)
0810 22:31:44.542746 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:44.542757 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:44.542911 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:44.543416 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:44.543419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:44.543437 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:44.543529 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.543535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.544746 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.544749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.546014 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.546023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.546795 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.546798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.546897 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.546931 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 22:31:44.547056 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 22:31:44.547178 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":70,"thread_start_us":79,"threads_started":1}
I20260810 22:31:44.547631 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.541503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.1.221.131:43589)
0810 22:31:44.541988 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:44.542008 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:44.542032 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:31:44.542129 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:44.542138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:44.542362 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:44.542531 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:44.543040 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.544186 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.545908 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.546706 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.546840 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.546976 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:44.547392 (+ 416us) server_negotiation.cc:299] Negotiation successful
0810 22:31:44.547491 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":130,"threads_started":1}
I20260810 22:31:44.547621 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.542365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36143 (local address 127.0.0.1:57168)
0810 22:31:44.543115 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:44.543128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:44.543211 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:44.543693 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:44.543696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:44.543713 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:44.543791 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.543800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.544925 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.544928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.546076 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.546084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.547279 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.547282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.547362 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.547382 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 22:31:44.547449 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 22:31:44.547504 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260810 22:31:44.547803 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.541303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:43589 (local address 127.0.0.1:55824)
0810 22:31:44.541789 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:44.541844 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:44.541951 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:44.542732 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:44.542738 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:44.542762 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:44.542911 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.542918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.544325 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.544328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.545773 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.545784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.547048 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.547053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.547146 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.547168 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:31:44.547235 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 22:31:44.547315 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":24,"thread_start_us":77,"threads_started":1}
I20260810 22:31:44.549243 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.542544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.1.221.130:36143)
0810 22:31:44.542859 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:44.542871 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:44.543309 (+ 438us) server_negotiation.cc:407] Connection header received
0810 22:31:44.543394 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:44.543399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:44.543567 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:44.543720 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:44.543886 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.544778 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.546208 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.546991 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.547091 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.547221 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:44.548890 (+ 1669us) server_negotiation.cc:299] Negotiation successful
0810 22:31:44.549097 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":128,"threads_started":1}
I20260810 22:31:44.550784 19392 tablet_service.cc:1467] Processing CreateTablet for tablet 6237948d8bf34059a516fa6ce0803a82 (DEFAULT_TABLE table=test-table [id=75d870828fdc46bfbc80633470446dc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:31:44.551141 19692 tablet_service.cc:1467] Processing CreateTablet for tablet 6237948d8bf34059a516fa6ce0803a82 (DEFAULT_TABLE table=test-table [id=75d870828fdc46bfbc80633470446dc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:31:44.551839 19392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6237948d8bf34059a516fa6ce0803a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:44.552155 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6237948d8bf34059a516fa6ce0803a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:44.551918 19541 tablet_service.cc:1467] Processing CreateTablet for tablet 6237948d8bf34059a516fa6ce0803a82 (DEFAULT_TABLE table=test-table [id=75d870828fdc46bfbc80633470446dc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:31:44.552862 19541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6237948d8bf34059a516fa6ce0803a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:31:44.558094 19786 tablet_bootstrap.cc:492] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Bootstrap starting.
I20260810 22:31:44.559393 19787 tablet_bootstrap.cc:492] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d: Bootstrap starting.
I20260810 22:31:44.560310 19788 tablet_bootstrap.cc:492] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc: Bootstrap starting.
I20260810 22:31:44.561741 19786 tablet_bootstrap.cc:654] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:44.562237 19787 tablet_bootstrap.cc:654] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:44.562528 19788 tablet_bootstrap.cc:654] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc: Neither blocks nor log segments found. Creating new log.
I20260810 22:31:44.563119 19787 log.cc:824] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:44.563119 19786 log.cc:824] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:44.563441 19788 log.cc:824] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc: Log is configured to *not* fsync() on all Append() calls
I20260810 22:31:44.565232 19786 tablet_bootstrap.cc:492] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: No bootstrap required, opened a new log
I20260810 22:31:44.565248 19788 tablet_bootstrap.cc:492] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc: No bootstrap required, opened a new log
I20260810 22:31:44.565441 19788 ts_tablet_manager.cc:1397] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:31:44.565441 19786 ts_tablet_manager.cc:1397] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260810 22:31:44.565701 19787 tablet_bootstrap.cc:492] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d: No bootstrap required, opened a new log
I20260810 22:31:44.565873 19787 ts_tablet_manager.cc:1397] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260810 22:31:44.568168 19788 raft_consensus.cc:348] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } }
I20260810 22:31:44.568413 19788 raft_consensus.cc:374] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:44.568468 19788 raft_consensus.cc:729] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0673f9c4c14dc9b244fda13157a1dc, State: Initialized, Role: FOLLOWER
I20260810 22:31:44.568943 19788 consensus_queue.cc:260] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } }
I20260810 22:31:44.569147 19786 raft_consensus.cc:348] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } }
I20260810 22:31:44.569432 19786 raft_consensus.cc:374] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:44.569491 19786 raft_consensus.cc:729] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef20c5cdc4342d895474b7513b901e4, State: Initialized, Role: FOLLOWER
I20260810 22:31:44.569340 19787 raft_consensus.cc:348] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } }
I20260810 22:31:44.569622 19787 raft_consensus.cc:374] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:31:44.569679 19787 raft_consensus.cc:729] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86aae5efd275476fa0cd52dd3b0edb9d, State: Initialized, Role: FOLLOWER
I20260810 22:31:44.569830 19621 heartbeater.cc:503] Master 127.1.221.190:44603 was elected leader, sending a full tablet report...
I20260810 22:31:44.569944 19786 consensus_queue.cc:260] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } }
I20260810 22:31:44.570065 19788 ts_tablet_manager.cc:1428] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260810 22:31:44.570075 19787 consensus_queue.cc:260] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } }
I20260810 22:31:44.570652 19469 heartbeater.cc:503] Master 127.1.221.190:44603 was elected leader, sending a full tablet report...
I20260810 22:31:44.570866 19787 ts_tablet_manager.cc:1428] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260810 22:31:44.570866 19786 ts_tablet_manager.cc:1428] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 22:31:44.571206 19767 heartbeater.cc:503] Master 127.1.221.190:44603 was elected leader, sending a full tablet report...
I20260810 22:31:44.637432 19792 raft_consensus.cc:482] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:31:44.637760 19792 raft_consensus.cc:504] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } }
I20260810 22:31:44.639026 19792 leader_election.cc:288] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589), 9ef20c5cdc4342d895474b7513b901e4 (127.1.221.129:40933)
I20260810 22:31:44.642796 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.639314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:43589 (local address 127.1.221.130:49237)
0810 22:31:44.639467 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:44.639481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:44.639591 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:44.639846 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:44.639849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:44.639865 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:44.639993 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.639999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.640820 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.640826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.641706 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.641715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.642372 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.642376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.642483 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.642505 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:31:44.642595 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 22:31:44.642662 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 22:31:44.642796 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.639470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:49237 (local address 127.1.221.131:43589)
0810 22:31:44.639610 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:44.639615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:44.639634 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:31:44.639692 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:44.639695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:44.639741 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:44.639833 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:44.640102 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.640708 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.641834 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.642241 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.642385 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.642440 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:44.642611 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 22:31:44.642683 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:31:44.643332 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.639633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:40933 (local address 127.1.221.130:43319)
0810 22:31:44.640047 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:44.640059 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:31:44.640174 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:31:44.640443 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:31:44.640446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:31:44.640461 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:31:44.640604 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.640610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.641384 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.641388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.642235 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:31:44.642257 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.642884 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.642888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:31:44.642989 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.643017 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 22:31:44.643098 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 22:31:44.643163 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":146,"threads_started":1}
I20260810 22:31:44.643337 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.639757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:43319 (local address 127.1.221.129:40933)
0810 22:31:44.639887 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:44.639891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:44.640143 (+ 252us) server_negotiation.cc:407] Connection header received
0810 22:31:44.640299 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:44.640304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:44.640359 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:44.640436 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:31:44.640715 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.641277 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.642368 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.642751 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.642874 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.642923 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:44.643133 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 22:31:44.643208 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:31:44.643512 19714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6237948d8bf34059a516fa6ce0803a82" candidate_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" is_pre_election: true
I20260810 22:31:44.643772 19714 raft_consensus.cc:2454] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b0673f9c4c14dc9b244fda13157a1dc in term 0.
I20260810 22:31:44.644230 19494 leader_election.cc:302] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86aae5efd275476fa0cd52dd3b0edb9d, 9b0673f9c4c14dc9b244fda13157a1dc; no voters:
I20260810 22:31:44.644526 19792 raft_consensus.cc:2781] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:31:44.644510 19417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6237948d8bf34059a516fa6ce0803a82" candidate_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef20c5cdc4342d895474b7513b901e4" is_pre_election: true
I20260810 22:31:44.644632 19792 raft_consensus.cc:482] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:31:44.644681 19792 raft_consensus.cc:3037] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:44.644820 19417 raft_consensus.cc:2454] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b0673f9c4c14dc9b244fda13157a1dc in term 0.
I20260810 22:31:44.645928 19792 raft_consensus.cc:504] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } }
I20260810 22:31:44.646220 19792 leader_election.cc:288] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [CANDIDATE]: Term 1 election: Requested vote from peers 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589), 9ef20c5cdc4342d895474b7513b901e4 (127.1.221.129:40933)
I20260810 22:31:44.646579 19714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6237948d8bf34059a516fa6ce0803a82" candidate_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86aae5efd275476fa0cd52dd3b0edb9d"
I20260810 22:31:44.646641 19417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6237948d8bf34059a516fa6ce0803a82" candidate_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef20c5cdc4342d895474b7513b901e4"
I20260810 22:31:44.646714 19714 raft_consensus.cc:3037] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:44.646780 19417 raft_consensus.cc:3037] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:31:44.647842 19714 raft_consensus.cc:2454] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0673f9c4c14dc9b244fda13157a1dc in term 1.
I20260810 22:31:44.647945 19417 raft_consensus.cc:2454] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0673f9c4c14dc9b244fda13157a1dc in term 1.
I20260810 22:31:44.648146 19494 leader_election.cc:302] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86aae5efd275476fa0cd52dd3b0edb9d, 9b0673f9c4c14dc9b244fda13157a1dc; no voters:
I20260810 22:31:44.648350 19792 raft_consensus.cc:2781] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:31:44.648617 19792 raft_consensus.cc:686] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [term 1 LEADER]: Becoming Leader. State: Replica: 9b0673f9c4c14dc9b244fda13157a1dc, State: Running, Role: LEADER
I20260810 22:31:44.648969 19792 consensus_queue.cc:237] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } }
I20260810 22:31:44.652158 19247 catalog_manager.cc:5495] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130). New cstate: current_term: 1 leader_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } health_report { overall_health: HEALTHY } } }
I20260810 22:31:44.670415 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.666822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.1.221.130:36143)
0810 22:31:44.666969 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:31:44.666974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:31:44.666988 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:31:44.667152 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:31:44.667154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:31:44.667214 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:31:44.667335 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:31:44.667567 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.668072 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.668912 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:31:44.669152 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:31:44.669261 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:31:44.669587 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:31:44.670006 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:31:44.670141 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 22:31:44.670205 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 22:31:44.670262 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:31:44.676597 19417 raft_consensus.cc:1261] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Refusing update from remote peer 9b0673f9c4c14dc9b244fda13157a1dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:31:44.676573 19714 raft_consensus.cc:1261] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d [term 1 FOLLOWER]: Refusing update from remote peer 9b0673f9c4c14dc9b244fda13157a1dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:31:44.677309 19792 consensus_queue.cc:1035] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:31:44.677526 19797 consensus_queue.cc:1035] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:31:44.689672 19803 mvcc.cc:205] Tried to move back new op lower bound from 7317098924747640832 to 7317098924644970496. Current Snapshot: MvccSnapshot[applied={T|T < 7317098924747640832 or (T in {7317098924747640832})}]
I20260810 22:31:44.695935 19245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:31:44.696108 19245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:31:44.698398 19245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Sending DeleteTablet for 3 replicas of tablet 6237948d8bf34059a516fa6ce0803a82
I20260810 22:31:44.699153 19692 tablet_service.cc:1514] Processing DeleteTablet for tablet 6237948d8bf34059a516fa6ce0803a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:44 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:55824
I20260810 22:31:44.699191 19392 tablet_service.cc:1514] Processing DeleteTablet for tablet 6237948d8bf34059a516fa6ce0803a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:44 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43028
I20260810 22:31:44.699324 19541 tablet_service.cc:1514] Processing DeleteTablet for tablet 6237948d8bf34059a516fa6ce0803a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:44 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:57168
I20260810 22:31:44.699631 19814 tablet_replica.cc:330] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d: stopping tablet replica
I20260810 22:31:44.699667 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 19320
I20260810 22:31:44.699770 19813 tablet_replica.cc:330] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: stopping tablet replica
I20260810 22:31:44.699965 19814 raft_consensus.cc:2229] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:31:44.700290 19814 raft_consensus.cc:2258] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:31:44.702567 19814 ts_tablet_manager.cc:1905] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:31:44.703773 19815 tablet_replica.cc:330] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc: stopping tablet replica
I20260810 22:31:44.704051 19815 raft_consensus.cc:2229] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:31:44.704455 19815 raft_consensus.cc:2258] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:31:44.705936 19815 ts_tablet_manager.cc:1905] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:31:44.706262 19814 ts_tablet_manager.cc:1918] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:31:44.706382 19814 log.cc:1196] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/6237948d8bf34059a516fa6ce0803a82
I20260810 22:31:44.706753 19814 ts_tablet_manager.cc:1939] T 6237948d8bf34059a516fa6ce0803a82 P 86aae5efd275476fa0cd52dd3b0edb9d: Deleting consensus metadata
I20260810 22:31:44.708103 19232 catalog_manager.cc:4841] TS 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589): tablet 6237948d8bf34059a516fa6ce0803a82 (table test-table [id=75d870828fdc46bfbc80633470446dc2]) successfully deleted
I20260810 22:31:44.708702 19815 ts_tablet_manager.cc:1918] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:31:44.708804 19815 log.cc:1196] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/6237948d8bf34059a516fa6ce0803a82
I20260810 22:31:44.709152 19815 ts_tablet_manager.cc:1939] T 6237948d8bf34059a516fa6ce0803a82 P 9b0673f9c4c14dc9b244fda13157a1dc: Deleting consensus metadata
I20260810 22:31:44.710271 19232 catalog_manager.cc:4841] TS 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130:36143): tablet 6237948d8bf34059a516fa6ce0803a82 (table test-table [id=75d870828fdc46bfbc80633470446dc2]) successfully deleted
W20260810 22:31:44.711030 19232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:40933 (error 108)
I20260810 22:31:44.711793 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 19474
I20260810 22:31:44.712040 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:31:44.711754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:40933 (local address 127.0.0.1:43044)
0810 22:31:44.711880 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 22:31:44.711949 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:40933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 22:31:44.714771 19232 catalog_manager.cc:4584] TS 9ef20c5cdc4342d895474b7513b901e4 (127.1.221.129:40933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6237948d8bf34059a516fa6ce0803a82: Network error: Client connection negotiation failed: client connection to 127.1.221.129:40933: connect: Connection refused (error 111)
I20260810 22:31:44.720391 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 19626
I20260810 22:31:44.727351 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 19213
I20260810 22:32:04.735703 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44603
--webserver_interface=127.1.221.190
--webserver_port=44949
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:44603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:04.849411 19817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:04.849706 19817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:04.849749 19817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:04.853781 19817 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 22:32:04.853832 19817 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 22:32:04.853847 19817 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 22:32:04.853860 19817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:32:04.853906 19817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:04.853945 19817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:04.853976 19817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:32:04.854015 19817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:32:04.859364 19817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:44603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=44949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:04.860463 19817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:04.861637 19817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:04.875398 19828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:04.875423 19824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:04.875595 19817 server_base.cc:1034] running on GCE node
W20260810 22:32:04.875789 19823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:04.876377 19817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:04.877481 19817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:04.878670 19817 hybrid_clock.cc:648] HybridClock initialized: now 1786401124878640 us; error 48 us; skew 500 ppm
Aug 10 22:32:04 dist-test-slave-njxd krb5kdc[19197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401124, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:04.881553 19817 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:32:04.882522 19817 webserver.cc:460] Webserver started at http://127.1.221.190:44949/ using document root <none> and password file <none>
I20260810 22:32:04.883117 19817 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:04.883229 19817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:04.887149 19817 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:32:04.889273 19835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:04.890578 19817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:04.890735 19817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "4d87813547aa4c75bc18d27acccaacf4"
format_stamp: "Formatted at 2026-08-10 22:31:43 on dist-test-slave-njxd"
server_key: "bdd61356327bcbeb314f278bc31abac2"
server_key_iv: "167df6b76ce225735ed0c403cc6595a2"
server_key_version: "encryptionkey@0"
I20260810 22:32:04.891324 19817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:04.911140 19817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:04.914795 19817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:04.915027 19817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:04.922938 19817 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:44603
I20260810 22:32:04.923251 19897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:44603 every 8 connection(s)
I20260810 22:32:04.924160 19817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:32:04.928606 19898 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:32:04.930387 19898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Bootstrap starting.
I20260810 22:32:04.933845 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 19817
I20260810 22:32:04.934351 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:40933
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:44603
--webserver_port=33749
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:32:04.950505 19898 log.cc:824] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:04.960016 19898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:32:04.960464 19898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Bootstrap complete.
I20260810 22:32:04.964820 19898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } }
I20260810 22:32:04.965225 19898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d87813547aa4c75bc18d27acccaacf4, State: Initialized, Role: FOLLOWER
I20260810 22:32:04.965781 19898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } }
I20260810 22:32:04.965905 19898 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:32:04.965999 19898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:32:04.966125 19898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:32:04.968070 19898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } }
I20260810 22:32:04.968417 19898 leader_election.cc:302] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d87813547aa4c75bc18d27acccaacf4; no voters:
I20260810 22:32:04.968750 19898 leader_election.cc:288] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 22:32:04.968901 19903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:32:04.969165 19903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 2 LEADER]: Becoming Leader. State: Replica: 4d87813547aa4c75bc18d27acccaacf4, State: Running, Role: LEADER
I20260810 22:32:04.969589 19903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } }
I20260810 22:32:04.970084 19898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:32:04.971001 19904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4d87813547aa4c75bc18d27acccaacf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } } }
I20260810 22:32:04.971122 19904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:04.971820 19905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d87813547aa4c75bc18d27acccaacf4. Latest consensus state: current_term: 2 leader_uuid: "4d87813547aa4c75bc18d27acccaacf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } } }
I20260810 22:32:04.971902 19905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:04.975175 19922 catalog_manager.cc:1259] Loaded cluster ID: a0357ccd43654cf294ad946082679dc0
I20260810 22:32:04.975238 19922 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: loading cluster ID for follower catalog manager: success
I20260810 22:32:04.977866 19922 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: acquiring CA information for follower catalog manager: success
I20260810 22:32:04.979403 19922 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:32:04.979565 19913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:32:04.982149 19913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:32:04.982748 19913 catalog_manager.cc:1259] Loaded cluster ID: a0357ccd43654cf294ad946082679dc0
I20260810 22:32:04.982805 19913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:32:04.983490 19913 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:32:04.983979 19913 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Loaded TSK: 0
I20260810 22:32:04.997475 19913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Generated new TSK 1
I20260810 22:32:04.997728 19913 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:32:05.051534 19900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:05.051867 19900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:05.051951 19900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:05.056183 19900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:05.056282 19900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:05.056377 19900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:32:05.061947 19900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:40933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=33749
--tserver_master_addrs=127.1.221.190:44603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:05.063021 19900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:05.064076 19900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:05.075284 19930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:05.075291 19932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:05.076098 19929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:05.076468 19900 server_base.cc:1034] running on GCE node
I20260810 22:32:05.077028 19900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:05.077749 19900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:05.078940 19900 hybrid_clock.cc:648] HybridClock initialized: now 1786401125078905 us; error 53 us; skew 500 ppm
Aug 10 22:32:05 dist-test-slave-njxd krb5kdc[19197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:05.081875 19900 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:32:05.082855 19900 webserver.cc:460] Webserver started at http://127.1.221.129:33749/ using document root <none> and password file <none>
I20260810 22:32:05.083439 19900 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:05.083513 19900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:05.087683 19900 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:32:05.089916 19940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:05.091310 19900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:32:05.091408 19900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "9ef20c5cdc4342d895474b7513b901e4"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "33af3b9be5af579681d6c77d0cc654ca"
server_key_iv: "58582788f6f70bdebb67d12569932197"
server_key_version: "encryptionkey@0"
I20260810 22:32:05.091989 19900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:05.101788 19900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:05.106808 19900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:05.107013 19900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:05.107681 19900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:05.108887 19947 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 22:32:05.113133 19900 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 22:32:05.113201 19900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 22:32:05.113253 19900 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 22:32:05.115087 19900 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 22:32:05.115151 19900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 22:32:05.115410 19947 tablet_bootstrap.cc:492] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Bootstrap starting.
I20260810 22:32:05.126662 19900 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:40933
I20260810 22:32:05.126747 20071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:40933 every 8 connection(s)
I20260810 22:32:05.128022 19900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:32:05.132364 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 19900
I20260810 22:32:05.132851 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:36143
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:44603
--webserver_port=40529
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:32:05.137912 19947 log.cc:824] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:32:05 dist-test-slave-njxd krb5kdc[19197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:32:05.146113 19947 tablet_bootstrap.cc:492] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:32:05.146588 19947 tablet_bootstrap.cc:492] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Bootstrap complete.
I20260810 22:32:05.148093 19947 ts_tablet_manager.cc:1397] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.008s
I20260810 22:32:05.150761 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.130294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:42159)
0810 22:32:05.131336 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:05.131377 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:05.132355 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:05.133814 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:05.133825 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:05.134372 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:05.135052 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.135069 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.136393 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.136396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.137273 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.137283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.138178 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.138183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.138407 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.141771 (+ 3364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:05.141793 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:05.143975 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:05.147006 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:05.147016 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:05.147034 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:05.147413 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:05.147873 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:05.147876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:05.147879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:05.148060 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:05.148825 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:05.148832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:05.149137 (+ 305us) client_negotiation.cc:769] Sending connection context
0810 22:32:05.149353 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 22:32:05.149748 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":180,"threads_started":1}
I20260810 22:32:05.150851 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.130300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:42159 (local address 127.1.221.190:44603)
0810 22:32:05.130716 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:05.130736 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:05.131500 (+ 764us) server_negotiation.cc:407] Connection header received
0810 22:32:05.132880 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:05.132896 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:05.133312 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:05.133752 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:05.135240 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.136217 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.137410 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.137960 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.138166 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.144271 (+ 6105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:05.144278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:05.144294 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:05.144332 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:05.146829 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:05.147553 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:05.147563 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:05.147571 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:05.147751 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:05.148181 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:05.148184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:05.148186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:05.148681 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:05.148815 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:05.149519 (+ 704us) server_negotiation.cc:299] Negotiation successful
0810 22:32:05.149975 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":139,"threads_started":1}
I20260810 22:32:05.151165 19947 raft_consensus.cc:348] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } }
I20260810 22:32:05.151665 19947 raft_consensus.cc:729] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef20c5cdc4342d895474b7513b901e4, State: Initialized, Role: FOLLOWER
I20260810 22:32:05.152195 19947 consensus_queue.cc:260] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } }
I20260810 22:32:05.153297 19947 ts_tablet_manager.cc:1428] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 22:32:05.154016 20072 heartbeater.cc:347] Connected to a master server at 127.1.221.190:44603
I20260810 22:32:05.154223 20072 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:05.154723 20072 heartbeater.cc:511] Master 127.1.221.190:44603 requested a full tablet report, sending...
I20260810 22:32:05.157384 19854 ts_manager.cc:194] Registered new tserver with Master: 9ef20c5cdc4342d895474b7513b901e4 (127.1.221.129:40933)
I20260810 22:32:05.162009 19854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:42159
I20260810 22:32:05.164700 20072 heartbeater.cc:503] Master 127.1.221.190:44603 was elected leader, sending a full tablet report...
Aug 10 22:32:05 dist-test-slave-njxd krb5kdc[19197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401124, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.129@KRBTEST.COM
I20260810 22:32:05.172390 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.160909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:40933 (local address 127.0.0.1:41698)
0810 22:32:05.161328 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:05.161371 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:05.161498 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:05.162160 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:05.162166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:05.162565 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:05.162673 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.162680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.164067 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.164071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.164996 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.165008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.165825 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.165828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.165964 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.166560 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:05.166575 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:05.168302 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:05.170445 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:05.170453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:05.170457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:05.170825 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:05.171185 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:05.171187 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:05.171189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:05.171273 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:05.171729 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:05.171735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:05.171857 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 22:32:05.172009 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 22:32:05.172207 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":132,"threads_started":1}
I20260810 22:32:05.172410 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.160960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.1.221.129:40933)
0810 22:32:05.161756 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:05.161766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:05.161783 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:32:05.161844 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:05.161847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:05.161996 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:05.162252 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:05.162842 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.163934 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.165147 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.165686 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.165788 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.168448 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:05.168458 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:05.168471 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:05.168498 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:05.170342 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:05.170976 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:05.170982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:05.170983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:05.171073 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:05.171366 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:05.171370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:05.171372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:05.171604 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:05.171792 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:05.172088 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 22:32:05.172260 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":704,"thread_start_us":118,"threads_started":1}
I20260810 22:32:05.173517 19975 tablet_service.cc:1514] Processing DeleteTablet for tablet 6237948d8bf34059a516fa6ce0803a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:31:44 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:41698
I20260810 22:32:05.174537 20081 tablet_replica.cc:330] stopping tablet replica
I20260810 22:32:05.174907 20081 raft_consensus.cc:2229] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:32:05.175179 20081 raft_consensus.cc:2258] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:05.176316 20081 ts_tablet_manager.cc:1905] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:05.181087 20081 ts_tablet_manager.cc:1918] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:05.181212 20081 log.cc:1196] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal/wals/6237948d8bf34059a516fa6ce0803a82
I20260810 22:32:05.181514 20081 ts_tablet_manager.cc:1939] T 6237948d8bf34059a516fa6ce0803a82 P 9ef20c5cdc4342d895474b7513b901e4: Deleting consensus metadata
I20260810 22:32:05.183594 19839 catalog_manager.cc:4841] TS 9ef20c5cdc4342d895474b7513b901e4 (127.1.221.129:40933): tablet 6237948d8bf34059a516fa6ce0803a82 (table test-table [id=75d870828fdc46bfbc80633470446dc2]) successfully deleted
W20260810 22:32:05.260689 20076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:05.261010 20076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:05.261108 20076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:05.265460 20076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:05.265529 20076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:05.265655 20076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:32:05.271449 20076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:36143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=40529
--tserver_master_addrs=127.1.221.190:44603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:05.272585 20076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:05.273648 20076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:05.283694 20086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:05.283687 20087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:05.283694 20089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:05.283970 20076 server_base.cc:1034] running on GCE node
I20260810 22:32:05.284574 20076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:05.285287 20076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:05.286489 20076 hybrid_clock.cc:648] HybridClock initialized: now 1786401125286469 us; error 34 us; skew 500 ppm
Aug 10 22:32:05 dist-test-slave-njxd krb5kdc[19197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:05.289498 20076 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:32:05.290601 20076 webserver.cc:460] Webserver started at http://127.1.221.130:40529/ using document root <none> and password file <none>
I20260810 22:32:05.291213 20076 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:05.291347 20076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:05.295783 20076 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:32:05.298234 20096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:05.299618 20076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:32:05.299785 20076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "9b0673f9c4c14dc9b244fda13157a1dc"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "8a36b533fd3a1367ffa681a5bdd26bda"
server_key_iv: "be37f9e940d5057fe9fdc325c196435b"
server_key_version: "encryptionkey@0"
I20260810 22:32:05.300390 20076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:05.327958 20076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:05.331122 20076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:05.331348 20076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:05.332049 20076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:05.332945 20076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:05.333019 20076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:05.333092 20076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:05.333139 20076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:05.342494 20076 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:36143
I20260810 22:32:05.342522 20225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:36143 every 8 connection(s)
I20260810 22:32:05.343596 20076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:32:05.353102 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 20076
I20260810 22:32:05.353632 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:43589
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:44603
--webserver_port=43219
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:32:05 dist-test-slave-njxd krb5kdc[19197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:32:05.358423 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.345548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:34569 (local address 127.1.221.190:44603)
0810 22:32:05.345736 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:05.345740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:05.346685 (+ 945us) server_negotiation.cc:407] Connection header received
0810 22:32:05.348071 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:05.348074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:05.348156 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:05.348261 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:05.349520 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.350121 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.351299 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.351551 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.351669 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.354366 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:05.354372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:05.354378 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:05.354416 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:05.356054 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:05.356691 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:05.356693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:05.356694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:05.356772 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:05.357218 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:05.357220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:05.357222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:05.357399 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:05.357477 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:05.358174 (+ 697us) server_negotiation.cc:299] Negotiation successful
0810 22:32:05.358297 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 22:32:05.359782 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.345955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:34569)
0810 22:32:05.346488 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:05.346534 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:05.347834 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:05.348455 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:05.348463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:05.348854 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:05.349350 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.349364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.350253 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.350258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.351163 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.351172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.351714 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.351718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.351892 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.352487 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:05.352503 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:05.354177 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:05.356182 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:05.356189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:05.356208 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:05.356587 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:05.356868 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:05.356872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:05.356877 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:05.357116 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:05.357509 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:05.357518 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:05.357889 (+ 371us) client_negotiation.cc:769] Sending connection context
0810 22:32:05.358122 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 22:32:05.358508 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":121,"threads_started":1}
I20260810 22:32:05.361341 20226 heartbeater.cc:347] Connected to a master server at 127.1.221.190:44603
I20260810 22:32:05.361677 20226 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:05.362380 20226 heartbeater.cc:511] Master 127.1.221.190:44603 requested a full tablet report, sending...
I20260810 22:32:05.363667 19854 ts_manager.cc:194] Registered new tserver with Master: 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130:36143)
I20260810 22:32:05.364595 19854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:34569
W20260810 22:32:05.474445 20229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:05.474727 20229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:05.474771 20229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:05.478889 20229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:05.478960 20229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:05.479022 20229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:32:05.484535 20229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:43589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=43219
--tserver_master_addrs=127.1.221.190:44603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:05.485723 20229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:05.486783 20229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:05.497457 20236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:05.497457 20235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:05.497460 20238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:05.497869 20229 server_base.cc:1034] running on GCE node
I20260810 22:32:05.498287 20229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:05.498981 20229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:05.500142 20229 hybrid_clock.cc:648] HybridClock initialized: now 1786401125500129 us; error 25 us; skew 500 ppm
Aug 10 22:32:05 dist-test-slave-njxd krb5kdc[19197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:05.503239 20229 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:32:05.504266 20229 webserver.cc:460] Webserver started at http://127.1.221.131:43219/ using document root <none> and password file <none>
I20260810 22:32:05.504914 20229 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:05.505026 20229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:05.508931 20229 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:05.511139 20245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:05.512346 20229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:32:05.512467 20229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "86aae5efd275476fa0cd52dd3b0edb9d"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "6edb581c84455be3b82c3660f767151f"
server_key_iv: "7f992e4d6d55f87c7faacb34dcdde0d8"
server_key_version: "encryptionkey@0"
I20260810 22:32:05.513012 20229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:05.539278 20229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:05.542955 20229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:05.543195 20229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:05.543874 20229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:05.544783 20229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:05.544837 20229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:05.544919 20229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:05.544953 20229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:05.554724 20229 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:43589
I20260810 22:32:05.554764 20376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:43589 every 8 connection(s)
I20260810 22:32:05.555850 20229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:32:05.562747 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 20229
Aug 10 22:32:05 dist-test-slave-njxd krb5kdc[19197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:32:05.571887 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.557937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:59655 (local address 127.1.221.190:44603)
0810 22:32:05.558126 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:05.558131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:05.558957 (+ 826us) server_negotiation.cc:407] Connection header received
0810 22:32:05.560287 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:05.560290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:05.560343 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:05.560428 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:05.561724 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.562273 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.563690 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.564043 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.564167 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.567059 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:05.567062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:05.567064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:05.567090 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:05.568879 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:05.569757 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:05.569760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:05.569762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:05.569901 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:05.570345 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:05.570349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:05.570351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:05.570639 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:05.570776 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:05.571520 (+ 744us) server_negotiation.cc:299] Negotiation successful
0810 22:32:05.571732 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"spinlock_wait_cycles":8448}
I20260810 22:32:05.572826 20382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.563588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.1.221.190:44603)
0810 22:32:05.563879 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:05.563883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:05.563996 (+ 113us) server_negotiation.cc:407] Connection header received
0810 22:32:05.564105 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:05.564110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:05.564167 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:05.564253 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:05.565250 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.565901 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.566847 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.567156 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.567237 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.568811 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:05.568815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:05.568817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:05.568862 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:05.570766 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:05.571214 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:05.571217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:05.571219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:05.571310 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:05.571645 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:05.571648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:05.571649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:05.571863 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:05.572079 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:05.572551 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 22:32:05.572681 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"spinlock_wait_cycles":11264,"thread_start_us":96,"threads_started":1}
I20260810 22:32:05.573001 20379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.558310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:59655)
0810 22:32:05.558789 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:05.558834 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:05.560043 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:05.560612 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:05.560619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:05.560999 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:05.561516 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.561530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.562387 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.562389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.563534 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.563546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.564240 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.564247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.564471 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.565122 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:05.565145 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:05.566873 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:05.569118 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:05.569127 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:05.569149 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:05.569573 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:05.570010 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:05.570015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:05.570018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:05.570218 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:05.570845 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:05.570854 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:05.571203 (+ 349us) client_negotiation.cc:769] Sending connection context
0810 22:32:05.571432 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 22:32:05.571839 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":165,"threads_started":1}
I20260810 22:32:05.574162 20377 heartbeater.cc:347] Connected to a master server at 127.1.221.190:44603
I20260810 22:32:05.574460 20377 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:05.575116 20377 heartbeater.cc:511] Master 127.1.221.190:44603 requested a full tablet report, sending...
I20260810 22:32:05.576197 19854 ts_manager.cc:194] Registered new tserver with Master: 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589)
I20260810 22:32:05.576741 19854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:59655
I20260810 22:32:05.587566 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:32:05.591770 20382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.588259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.1.221.190:44603)
0810 22:32:05.588394 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:05.588397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:05.588557 (+ 160us) server_negotiation.cc:407] Connection header received
0810 22:32:05.588718 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:05.588720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:05.588826 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:05.588917 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:32:05.589144 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.589782 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.590657 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.590903 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.590984 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.591269 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:32:05.591420 (+ 151us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 22:32:05.591642 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58144: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 22:32:05.591869 20382 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58144: token expired
W20260810 22:32:05.592186 19770 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.221.190:44603)
I20260810 22:32:05.599409 20382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.592566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.1.221.190:44603)
0810 22:32:05.592690 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:05.592693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:05.592705 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:32:05.592770 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:05.592773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:05.592810 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:05.592873 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:05.593471 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.594007 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.594840 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.595040 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.595118 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.596617 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:05.596621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:05.596624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:05.596651 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:05.598037 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:05.598426 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:05.598429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:05.598430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:05.598503 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:05.598733 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:05.598735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:05.598736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:05.598899 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:05.598997 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:05.599196 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 22:32:05.599291 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:32:05.601886 19854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:32:05.605268 19854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:32:05.613312 19975 tablet_service.cc:1467] Processing CreateTablet for tablet 2a8fa7445b9b414bb589676e94245087 (DEFAULT_TABLE table=test-table [id=f765e239a98948798c8f840fa73878c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:05.614029 19975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a8fa7445b9b414bb589676e94245087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:05.617165 19947 tablet_bootstrap.cc:492] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4: Bootstrap starting.
I20260810 22:32:05.617938 20387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.612269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36143 (local address 127.0.0.1:53836)
0810 22:32:05.612550 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:05.612564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:05.612660 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:05.614013 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:05.614016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:05.614034 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:05.614138 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.614144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.615428 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.615433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.616220 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.616229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.617444 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.617448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.617541 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.617634 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 22:32:05.617712 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 22:32:05.617792 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":152,"threads_started":1}
I20260810 22:32:05.618057 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.612870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.1.221.130:36143)
0810 22:32:05.613246 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:05.613258 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:05.613282 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:32:05.613362 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:05.613369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:05.613621 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:05.613808 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:05.614281 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.615311 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.616430 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.617317 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.617534 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.617728 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:05.617787 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 22:32:05.617908 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":128,"threads_started":1}
I20260810 22:32:05.618256 19947 tablet_bootstrap.cc:654] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:05.618347 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.611998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:43589 (local address 127.0.0.1:38524)
0810 22:32:05.612239 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:05.612257 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:05.612344 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:05.613185 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:05.613189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:05.613207 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:05.613322 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.613328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.615044 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.615048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.616281 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.616292 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.617926 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.617930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.618043 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.618075 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 22:32:05.618131 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 22:32:05.618231 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260810 22:32:05.618974 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.612215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.1.221.131:43589)
0810 22:32:05.612602 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:05.612617 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:05.612646 (+ 29us) server_negotiation.cc:407] Connection header received
0810 22:32:05.612740 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:05.612746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:05.612979 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:05.613486 (+ 507us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:05.613708 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.614913 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.616429 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.617797 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.618630 (+ 833us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.618747 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:05.618788 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 22:32:05.618863 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":184,"threads_started":1}
I20260810 22:32:05.620002 19947 tablet_bootstrap.cc:492] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4: No bootstrap required, opened a new log
I20260810 22:32:05.620117 19947 ts_tablet_manager.cc:1397] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:32:05.620524 19947 raft_consensus.cc:348] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:05.620648 19947 raft_consensus.cc:374] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:05.620707 19947 raft_consensus.cc:729] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef20c5cdc4342d895474b7513b901e4, State: Initialized, Role: FOLLOWER
I20260810 22:32:05.620842 19947 consensus_queue.cc:260] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:05.621131 19947 ts_tablet_manager.cc:1428] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:32:05.622186 20297 tablet_service.cc:1467] Processing CreateTablet for tablet 2a8fa7445b9b414bb589676e94245087 (DEFAULT_TABLE table=test-table [id=f765e239a98948798c8f840fa73878c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:05.623406 20297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a8fa7445b9b414bb589676e94245087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:05.623785 20149 tablet_service.cc:1467] Processing CreateTablet for tablet 2a8fa7445b9b414bb589676e94245087 (DEFAULT_TABLE table=test-table [id=f765e239a98948798c8f840fa73878c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:05.624845 20149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a8fa7445b9b414bb589676e94245087. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 22:32:05.629565 20073 tablet.cc:2369] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:32:05.632203 20391 tablet_bootstrap.cc:492] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d: Bootstrap starting.
I20260810 22:32:05.632849 20392 tablet_bootstrap.cc:492] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc: Bootstrap starting.
I20260810 22:32:05.635511 20392 tablet_bootstrap.cc:654] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:05.635511 20391 tablet_bootstrap.cc:654] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:05.636509 20392 log.cc:824] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:05.636509 20391 log.cc:824] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:05.638993 20392 tablet_bootstrap.cc:492] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc: No bootstrap required, opened a new log
I20260810 22:32:05.639019 20391 tablet_bootstrap.cc:492] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d: No bootstrap required, opened a new log
I20260810 22:32:05.639218 20391 ts_tablet_manager.cc:1397] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260810 22:32:05.639218 20392 ts_tablet_manager.cc:1397] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 22:32:05.643386 20392 raft_consensus.cc:348] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:05.643391 20391 raft_consensus.cc:348] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:05.643702 20391 raft_consensus.cc:374] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:05.643707 20392 raft_consensus.cc:374] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:05.643766 20392 raft_consensus.cc:729] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0673f9c4c14dc9b244fda13157a1dc, State: Initialized, Role: FOLLOWER
I20260810 22:32:05.643766 20391 raft_consensus.cc:729] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86aae5efd275476fa0cd52dd3b0edb9d, State: Initialized, Role: FOLLOWER
I20260810 22:32:05.644270 20392 consensus_queue.cc:260] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:05.644270 20391 consensus_queue.cc:260] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:05.645016 20226 heartbeater.cc:503] Master 127.1.221.190:44603 was elected leader, sending a full tablet report...
I20260810 22:32:05.645265 20377 heartbeater.cc:503] Master 127.1.221.190:44603 was elected leader, sending a full tablet report...
I20260810 22:32:05.645401 20392 ts_tablet_manager.cc:1428] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 22:32:05.646339 20391 ts_tablet_manager.cc:1428] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
W20260810 22:32:05.806893 20378 tablet.cc:2369] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:32:05.844841 20227 tablet.cc:2369] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:32:05.871963 20395 raft_consensus.cc:482] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:32:05.872169 20395 raft_consensus.cc:504] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:05.873353 20395 leader_election.cc:288] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589), 9ef20c5cdc4342d895474b7513b901e4 (127.1.221.129:40933)
I20260810 22:32:05.878001 20388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.873900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:45387 (local address 127.1.221.131:43589)
0810 22:32:05.874048 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:05.874052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:05.874232 (+ 180us) server_negotiation.cc:407] Connection header received
0810 22:32:05.874425 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:05.874429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:05.874492 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:05.874592 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:05.875063 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.875662 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.876964 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.877398 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.877495 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.877572 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:05.877792 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 22:32:05.877880 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:32:05.878005 20397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.873801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:43589 (local address 127.1.221.130:45387)
0810 22:32:05.874118 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:05.874136 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:05.874262 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:05.874765 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:05.874770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:05.874792 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:05.874930 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.874941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.875781 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.875785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.876823 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.876833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.877524 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.877530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.877654 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.877677 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:32:05.877755 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 22:32:05.877834 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":31,"thread_start_us":124,"threads_started":1}
I20260810 22:32:05.878074 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.874401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:36071 (local address 127.1.221.129:40933)
0810 22:32:05.874756 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:05.874761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:05.874773 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:32:05.874825 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:05.874828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:05.874891 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:05.875014 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:05.875319 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.875904 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.877018 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.877465 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.877621 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.877787 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:05.877871 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 22:32:05.877942 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":113,"threads_started":1}
I20260810 22:32:05.878485 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.874240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:40933 (local address 127.1.221.130:36071)
0810 22:32:05.874487 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:05.874504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:05.874650 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:05.875049 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:05.875052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:05.875071 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:05.875202 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.875209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.876017 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.876019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.876849 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:05.876859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.877636 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.877639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:05.877721 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.877770 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 22:32:05.877854 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 22:32:05.877919 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":35,"thread_start_us":88,"threads_started":1}
I20260810 22:32:05.878715 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a8fa7445b9b414bb589676e94245087" candidate_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" is_pre_election: true
I20260810 22:32:05.879030 20320 raft_consensus.cc:2454] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b0673f9c4c14dc9b244fda13157a1dc in term 0.
I20260810 22:32:05.879051 20012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a8fa7445b9b414bb589676e94245087" candidate_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef20c5cdc4342d895474b7513b901e4" is_pre_election: true
I20260810 22:32:05.879361 20012 raft_consensus.cc:2454] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b0673f9c4c14dc9b244fda13157a1dc in term 0.
I20260810 22:32:05.879596 20099 leader_election.cc:302] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86aae5efd275476fa0cd52dd3b0edb9d, 9b0673f9c4c14dc9b244fda13157a1dc; no voters:
I20260810 22:32:05.879891 20395 raft_consensus.cc:2781] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:32:05.879967 20395 raft_consensus.cc:482] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:32:05.879997 20395 raft_consensus.cc:3037] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:05.882834 20395 raft_consensus.cc:504] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:05.883199 20395 leader_election.cc:288] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [CANDIDATE]: Term 1 election: Requested vote from peers 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589), 9ef20c5cdc4342d895474b7513b901e4 (127.1.221.129:40933)
I20260810 22:32:05.883649 20320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a8fa7445b9b414bb589676e94245087" candidate_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86aae5efd275476fa0cd52dd3b0edb9d"
I20260810 22:32:05.883677 20012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a8fa7445b9b414bb589676e94245087" candidate_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef20c5cdc4342d895474b7513b901e4"
I20260810 22:32:05.883809 20320 raft_consensus.cc:3037] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:05.883828 20012 raft_consensus.cc:3037] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:05.884863 20012 raft_consensus.cc:2454] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0673f9c4c14dc9b244fda13157a1dc in term 1.
I20260810 22:32:05.884912 20320 raft_consensus.cc:2454] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b0673f9c4c14dc9b244fda13157a1dc in term 1.
I20260810 22:32:05.885146 20099 leader_election.cc:302] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b0673f9c4c14dc9b244fda13157a1dc, 9ef20c5cdc4342d895474b7513b901e4; no voters:
I20260810 22:32:05.885353 20395 raft_consensus.cc:2781] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:05.885645 20395 raft_consensus.cc:686] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [term 1 LEADER]: Becoming Leader. State: Replica: 9b0673f9c4c14dc9b244fda13157a1dc, State: Running, Role: LEADER
I20260810 22:32:05.886003 20395 consensus_queue.cc:237] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:05.889225 19853 catalog_manager.cc:5495] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130). New cstate: current_term: 1 leader_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } health_report { overall_health: UNKNOWN } } }
I20260810 22:32:05.914430 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.910577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.1.221.130:36143)
0810 22:32:05.910723 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:05.910731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:05.910785 (+ 54us) server_negotiation.cc:407] Connection header received
0810 22:32:05.911008 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:05.911011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:05.911082 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:05.911202 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:32:05.911481 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.912000 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.912895 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:05.913129 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:05.913245 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:05.913591 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:32:05.914010 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:32:05.914154 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:05.914216 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 22:32:05.914279 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 22:32:05.920789 20320 raft_consensus.cc:1261] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d [term 1 FOLLOWER]: Refusing update from remote peer 9b0673f9c4c14dc9b244fda13157a1dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:05.920833 20012 raft_consensus.cc:1261] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Refusing update from remote peer 9b0673f9c4c14dc9b244fda13157a1dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:05.921546 20400 consensus_queue.cc:1035] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:32:05.921842 20395 consensus_queue.cc:1035] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:32:05.929941 20404 mvcc.cc:205] Tried to move back new op lower bound from 7317099011764031488 to 7317099011631575040. Current Snapshot: MvccSnapshot[applied={T|T < 7317099011764031488}]
I20260810 22:32:05.932520 20403 mvcc.cc:205] Tried to move back new op lower bound from 7317099011764031488 to 7317099011631575040. Current Snapshot: MvccSnapshot[applied={T|T < 7317099011764031488}]
I20260810 22:32:05.932762 20402 mvcc.cc:205] Tried to move back new op lower bound from 7317099011764031488 to 7317099011631575040. Current Snapshot: MvccSnapshot[applied={T|T < 7317099011764031488}]
I20260810 22:32:05.942412 19853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:05.942613 19853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:05.945022 19853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Sending DeleteTablet for 3 replicas of tablet 2a8fa7445b9b414bb589676e94245087
I20260810 22:32:05.945924 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 19900
I20260810 22:32:05.945873 20297 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a8fa7445b9b414bb589676e94245087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:05 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:38524
I20260810 22:32:05.946161 20149 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a8fa7445b9b414bb589676e94245087 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:05 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:53836
I20260810 22:32:05.949811 20414 tablet_replica.cc:330] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc: stopping tablet replica
I20260810 22:32:05.950371 20414 raft_consensus.cc:2229] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:32:05.951038 20414 raft_consensus.cc:2258] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:05.953676 20414 ts_tablet_manager.cc:1905] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:05.953871 20413 tablet_replica.cc:330] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d: stopping tablet replica
I20260810 22:32:05.954216 20413 raft_consensus.cc:2229] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:32:05.954602 20413 raft_consensus.cc:2258] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:05.956555 20414 ts_tablet_manager.cc:1918] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:05.956682 20414 log.cc:1196] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/2a8fa7445b9b414bb589676e94245087
I20260810 22:32:05.957057 20414 ts_tablet_manager.cc:1939] T 2a8fa7445b9b414bb589676e94245087 P 9b0673f9c4c14dc9b244fda13157a1dc: Deleting consensus metadata
I20260810 22:32:05.958288 20413 ts_tablet_manager.cc:1905] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:05.958482 19839 catalog_manager.cc:4841] TS 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130:36143): tablet 2a8fa7445b9b414bb589676e94245087 (table test-table [id=f765e239a98948798c8f840fa73878c1]) successfully deleted
I20260810 22:32:05.960278 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 20076
I20260810 22:32:05.961448 20413 ts_tablet_manager.cc:1918] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:05.961628 20413 log.cc:1196] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/2a8fa7445b9b414bb589676e94245087
I20260810 22:32:05.962029 20413 ts_tablet_manager.cc:1939] T 2a8fa7445b9b414bb589676e94245087 P 86aae5efd275476fa0cd52dd3b0edb9d: Deleting consensus metadata
W20260810 22:32:05.963186 19839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:40933 (error 108)
I20260810 22:32:05.963793 19839 catalog_manager.cc:4841] TS 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589): tablet 2a8fa7445b9b414bb589676e94245087 (table test-table [id=f765e239a98948798c8f840fa73878c1]) successfully deleted
I20260810 22:32:05.964413 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:05.964131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:40933 (local address 127.0.0.1:41706)
0810 22:32:05.964274 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:05.964341 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:40933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 22:32:05.964566 19839 catalog_manager.cc:4584] TS 9ef20c5cdc4342d895474b7513b901e4 (127.1.221.129:40933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a8fa7445b9b414bb589676e94245087: Network error: Client connection negotiation failed: client connection to 127.1.221.129:40933: connect: Connection refused (error 111)
I20260810 22:32:05.968433 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 20229
I20260810 22:32:05.975651 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 19817
I20260810 22:32:25.984522 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44603
--webserver_interface=127.1.221.190
--webserver_port=44949
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:44603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:26.100981 20417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:26.101253 20417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:26.101325 20417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:26.105265 20417 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 22:32:26.105337 20417 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 22:32:26.105365 20417 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 22:32:26.105400 20417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:32:26.105440 20417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:26.105480 20417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:26.105516 20417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:32:26.105552 20417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:32:26.110677 20417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:44603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=44949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:26.111735 20417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:26.112746 20417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:26.121730 20425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:26.121699 20422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:26.121699 20423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:26.121754 20417 server_base.cc:1034] running on GCE node
I20260810 22:32:26.122632 20417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:26.123740 20417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:26.124931 20417 hybrid_clock.cc:648] HybridClock initialized: now 1786401146124918 us; error 29 us; skew 500 ppm
Aug 10 22:32:26 dist-test-slave-njxd krb5kdc[19197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401146, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:26.131601 20417 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:32:26.132469 20417 webserver.cc:460] Webserver started at http://127.1.221.190:44949/ using document root <none> and password file <none>
I20260810 22:32:26.133002 20417 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:26.133062 20417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:26.136973 20417 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:26.139272 20433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:26.140709 20417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:32:26.140817 20417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "4d87813547aa4c75bc18d27acccaacf4"
format_stamp: "Formatted at 2026-08-10 22:31:43 on dist-test-slave-njxd"
server_key: "bdd61356327bcbeb314f278bc31abac2"
server_key_iv: "167df6b76ce225735ed0c403cc6595a2"
server_key_version: "encryptionkey@0"
I20260810 22:32:26.141364 20417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:26.155488 20417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:26.159667 20417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:26.159881 20417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:26.170429 20417 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:44603
I20260810 22:32:26.170702 20495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:44603 every 8 connection(s)
I20260810 22:32:26.171612 20417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:32:26.172806 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 20417
I20260810 22:32:26.173314 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:40933
--local_ip_for_outbound_sockets=127.1.221.129
--tserver_master_addrs=127.1.221.190:44603
--webserver_port=33749
--webserver_interface=127.1.221.129
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:32:26.177150 20496 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:32:26.179562 20496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Bootstrap starting.
I20260810 22:32:26.197924 20496 log.cc:824] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:26.208668 20496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:32:26.209054 20496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Bootstrap complete.
I20260810 22:32:26.213191 20496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } }
I20260810 22:32:26.213550 20496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d87813547aa4c75bc18d27acccaacf4, State: Initialized, Role: FOLLOWER
I20260810 22:32:26.214079 20496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } }
I20260810 22:32:26.214192 20496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:32:26.214282 20496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:32:26.214391 20496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 2 FOLLOWER]: Advancing to term 3
I20260810 22:32:26.216218 20496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } }
I20260810 22:32:26.216537 20496 leader_election.cc:302] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d87813547aa4c75bc18d27acccaacf4; no voters:
I20260810 22:32:26.216881 20496 leader_election.cc:288] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 22:32:26.216960 20501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 22:32:26.217250 20501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [term 3 LEADER]: Becoming Leader. State: Replica: 4d87813547aa4c75bc18d27acccaacf4, State: Running, Role: LEADER
I20260810 22:32:26.217617 20501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } }
I20260810 22:32:26.218093 20496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:32:26.219007 20502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4d87813547aa4c75bc18d27acccaacf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } } }
I20260810 22:32:26.219138 20502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:26.219708 20503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d87813547aa4c75bc18d27acccaacf4. Latest consensus state: current_term: 3 leader_uuid: "4d87813547aa4c75bc18d27acccaacf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d87813547aa4c75bc18d27acccaacf4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44603 } } }
I20260810 22:32:26.219852 20503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:26.220158 20511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:32:26.225258 20511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:32:26.225921 20511 catalog_manager.cc:1259] Loaded cluster ID: a0357ccd43654cf294ad946082679dc0
I20260810 22:32:26.225983 20511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:32:26.229023 20511 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:32:26.229864 20511 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Loaded TSK: 0
I20260810 22:32:26.229965 20511 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d87813547aa4c75bc18d27acccaacf4: Loaded TSK: 1
F20260810 22:32:26.230756 20511 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f604c2149ad google::LogMessage::Fail() at ??:0
@ 0x7f604c21788a google::LogMessage::SendToLog() at ??:0
@ 0x7f604c2144de google::LogMessage::Flush() at ??:0
@ 0x7f604c215a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f604eed818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f604eed0f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6056b797bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6056b5399c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6056b54178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f6056b95df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f6056bb482b std::function<>::operator()() at ??:0
@ 0x7f6056b53b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6056b54f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6056b51579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6056b95640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f604e4663d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f604e466c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f604e4683f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f604e457512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f604fa9b6db start_thread at ??:0
@ 0x7f604ad1261f clone at ??:0
W20260810 22:32:26.332448 20499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:26.332710 20499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:26.332784 20499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:26.337020 20499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:26.337076 20499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:26.337167 20499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:32:26.342839 20499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:40933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=33749
--tserver_master_addrs=127.1.221.190:44603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:26.343957 20499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:26.345018 20499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:26.354959 20549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:26.354959 20548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:26.355145 20499 server_base.cc:1034] running on GCE node
W20260810 22:32:26.355254 20551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:26.355652 20499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:26.356315 20499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:26.357497 20499 hybrid_clock.cc:648] HybridClock initialized: now 1786401146357476 us; error 38 us; skew 500 ppm
Aug 10 22:32:26 dist-test-slave-njxd krb5kdc[19197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401146, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:26.360563 20499 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:32:26.361493 20499 webserver.cc:460] Webserver started at http://127.1.221.129:33749/ using document root <none> and password file <none>
I20260810 22:32:26.362108 20499 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:26.362182 20499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:26.366196 20499 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:32:26.368510 20558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:26.369900 20499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:32:26.370030 20499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "9ef20c5cdc4342d895474b7513b901e4"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "33af3b9be5af579681d6c77d0cc654ca"
server_key_iv: "58582788f6f70bdebb67d12569932197"
server_key_version: "encryptionkey@0"
I20260810 22:32:26.370579 20499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:26.385265 20499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:26.388510 20499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:26.388713 20499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:26.389295 20499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:26.390461 20565 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 22:32:26.395753 20499 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 22:32:26.395828 20499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 22:32:26.395906 20499 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 22:32:26.397502 20499 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 22:32:26.397567 20499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 22:32:26.397868 20565 tablet_bootstrap.cc:492] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4: Bootstrap starting.
I20260810 22:32:26.408419 20499 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:40933
I20260810 22:32:26.408612 20685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:40933 every 8 connection(s)
I20260810 22:32:26.409370 20499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:32:26.412500 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 20499
I20260810 22:32:26.413022 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:36143
--local_ip_for_outbound_sockets=127.1.221.130
--tserver_master_addrs=127.1.221.190:44603
--webserver_port=40529
--webserver_interface=127.1.221.130
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:32:26.421288 20565 log.cc:824] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:26.431205 20565 tablet_bootstrap.cc:492] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:32:26.431708 20565 tablet_bootstrap.cc:492] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4: Bootstrap complete.
I20260810 22:32:26.433409 20565 ts_tablet_manager.cc:1397] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4: Time spent bootstrapping tablet: real 0.036s user 0.020s sys 0.011s
I20260810 22:32:26.436890 20565 raft_consensus.cc:348] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:26.437446 20565 raft_consensus.cc:729] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef20c5cdc4342d895474b7513b901e4, State: Initialized, Role: FOLLOWER
I20260810 22:32:26.438036 20565 consensus_queue.cc:260] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:26.439127 20565 ts_tablet_manager.cc:1428] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260810 22:32:26.477685 20688 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:32:26.411314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:49541)
0810 22:32:26.411800 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:26.411853 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:26.413029 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:26.476753 (+ 63724us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":157,"threads_started":1}
W20260810 22:32:26.478400 20686 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:44603 (0 consecutive failures): Network error: Failed to ping master at 127.1.221.190:44603: Client connection negotiation failed: client connection to 127.1.221.190:44603: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 22:32:26.479295 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:26.478969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:40435)
0810 22:32:26.479165 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:26.479233 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260810 22:32:26.479967 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:26.479754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:52085)
0810 22:32:26.479863 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:26.479917 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 22:32:26.480165 20686 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:32:26.534904 20689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:26.535252 20689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:26.535341 20689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:26.539472 20689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:26.539569 20689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:26.539659 20689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:32:26.545047 20689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:36143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=40529
--tserver_master_addrs=127.1.221.190:44603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:26.546293 20689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:26.547369 20689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:26.557319 20700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:26.557327 20698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:26.557330 20697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:26.557516 20689 server_base.cc:1034] running on GCE node
I20260810 22:32:26.557991 20689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:26.558707 20689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:26.559901 20689 hybrid_clock.cc:648] HybridClock initialized: now 1786401146559888 us; error 32 us; skew 500 ppm
Aug 10 22:32:26 dist-test-slave-njxd krb5kdc[19197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401146, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:26.562863 20689 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:32:26.563720 20689 webserver.cc:460] Webserver started at http://127.1.221.130:40529/ using document root <none> and password file <none>
I20260810 22:32:26.564266 20689 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:26.564327 20689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:26.568159 20689 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:26.570225 20708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:26.571446 20689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:26.571540 20689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "9b0673f9c4c14dc9b244fda13157a1dc"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "8a36b533fd3a1367ffa681a5bdd26bda"
server_key_iv: "be37f9e940d5057fe9fdc325c196435b"
server_key_version: "encryptionkey@0"
I20260810 22:32:26.572057 20689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:26.597561 20689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:26.600947 20689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:26.601102 20689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:26.601684 20689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:26.602463 20689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:26.602509 20689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:26.602550 20689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:26.602564 20689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:26.611726 20689 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:36143
I20260810 22:32:26.611768 20836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:36143 every 8 connection(s)
I20260810 22:32:26.612746 20689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:32:26.616717 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:26.614887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:48887)
0810 22:32:26.615348 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:26.615457 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":123,"threads_started":1}
W20260810 22:32:26.617165 20837 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:44603 (0 consecutive failures): Network error: Failed to ping master at 127.1.221.190:44603: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
I20260810 22:32:26.617805 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:26.617559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:50435)
0810 22:32:26.617693 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:26.617749 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 22:32:26.618404 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:26.618194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:46067)
0810 22:32:26.618303 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:26.618349 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 22:32:26.618618 20837 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 22:32:26.622881 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 20689
I20260810 22:32:26.623322 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:43589
--local_ip_for_outbound_sockets=127.1.221.131
--tserver_master_addrs=127.1.221.190:44603
--webserver_port=43219
--webserver_interface=127.1.221.131
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:26.739884 20840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:26.740170 20840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:26.740243 20840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:26.744536 20840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:26.744617 20840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:26.744717 20840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:32:26.750221 20840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:43589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=43219
--tserver_master_addrs=127.1.221.190:44603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:26.751370 20840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:26.752491 20840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:26.762494 20848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:26.762452 20846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:26.762490 20845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:26.762669 20840 server_base.cc:1034] running on GCE node
I20260810 22:32:26.763331 20840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:26.764077 20840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:26.765271 20840 hybrid_clock.cc:648] HybridClock initialized: now 1786401146765251 us; error 37 us; skew 500 ppm
Aug 10 22:32:26 dist-test-slave-njxd krb5kdc[19197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401146, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:26.768074 20840 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:32:26.769001 20840 webserver.cc:460] Webserver started at http://127.1.221.131:43219/ using document root <none> and password file <none>
I20260810 22:32:26.769654 20840 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:26.769752 20840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:26.773746 20840 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:32:26.776000 20857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:26.777391 20840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:32:26.777491 20840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "86aae5efd275476fa0cd52dd3b0edb9d"
format_stamp: "Formatted at 2026-08-10 22:31:44 on dist-test-slave-njxd"
server_key: "6edb581c84455be3b82c3660f767151f"
server_key_iv: "7f992e4d6d55f87c7faacb34dcdde0d8"
server_key_version: "encryptionkey@0"
I20260810 22:32:26.778110 20840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:26.790542 20840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:26.793503 20840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:26.793727 20840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:26.794296 20840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:26.795099 20840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:26.795145 20840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:26.795185 20840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:26.795200 20840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:26.804345 20840 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:43589
I20260810 22:32:26.804435 20988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:43589 every 8 connection(s)
I20260810 22:32:26.805434 20840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:32:26.809437 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:26.807644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:56223)
0810 22:32:26.808115 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:26.808238 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
W20260810 22:32:26.809944 20989 heartbeater.cc:644] Failed to heartbeat to 127.1.221.190:44603 (0 consecutive failures): Network error: Failed to ping master at 127.1.221.190:44603: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
I20260810 22:32:26.810609 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:26.810352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:42627)
0810 22:32:26.810474 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:26.810551 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 22:32:26.810850 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 20840
I20260810 22:32:26.811327 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:26.811112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:58115)
0810 22:32:26.811222 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:26.811268 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 22:32:26.811580 20989 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 22:32:27.481410 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:27.480834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:48719)
0810 22:32:27.481139 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:27.481205 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":85,"threads_started":1}
I20260810 22:32:27.619668 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:27.619249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:40933)
0810 22:32:27.619502 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:27.619562 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":69,"threads_started":1}
I20260810 22:32:27.812492 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:27.812106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:53813)
0810 22:32:27.812342 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:27.812393 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260810 22:32:28.108912 20997 raft_consensus.cc:482] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:32:28.109148 20997 raft_consensus.cc:504] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:28.110306 20997 leader_election.cc:288] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589), 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130:36143)
Aug 10 22:32:28 dist-test-slave-njxd krb5kdc[19197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401146, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.130@KRBTEST.COM
Aug 10 22:32:28 dist-test-slave-njxd krb5kdc[19197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401146, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.131@KRBTEST.COM
I20260810 22:32:28.126164 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:28.111132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:36143 (local address 127.1.221.129:43357)
0810 22:32:28.111446 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:28.111461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:28.111571 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:28.113043 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:28.113054 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:28.113694 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:28.114258 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:28.114275 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:28.115731 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:28.115735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:28.116943 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:28.116953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:28.117784 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:28.117787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:28.117933 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:28.118629 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:28.118652 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:28.120740 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:28.123215 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:28.123224 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:28.123242 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:28.123510 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:28.123850 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:28.123854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:28.123856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:28.124014 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:28.125029 (+ 1015us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:28.125034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:28.125386 (+ 352us) client_negotiation.cc:769] Sending connection context
0810 22:32:28.125634 (+ 248us) client_negotiation.cc:240] Negotiation successful
0810 22:32:28.125970 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":174,"spinlock_wait_cycles":23296,"thread_start_us":83,"threads_started":1}
I20260810 22:32:28.126693 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:28.110727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:43589 (local address 127.1.221.129:44687)
0810 22:32:28.111001 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:28.111019 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:28.111138 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:28.113044 (+ 1906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:28.113051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:28.113692 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:28.114258 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:28.114274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:28.115728 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:28.115732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:28.116939 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:28.116950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:28.118589 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:28.118593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:28.118725 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:28.119142 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:28.119157 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:28.121855 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:28.124433 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:28.124436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:28.124438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:28.124674 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:28.125009 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:28.125011 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:28.125012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:28.125089 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:28.125881 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:28.125885 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:28.125979 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 22:32:28.126094 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 22:32:28.126232 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":71,"thread_start_us":111,"threads_started":1}
I20260810 22:32:28.126714 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:28.111358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:43357 (local address 127.1.221.130:36143)
0810 22:32:28.111686 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:28.111706 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:28.111740 (+ 34us) server_negotiation.cc:407] Connection header received
0810 22:32:28.112406 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:28.112412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:28.112733 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:28.113045 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:28.114456 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:28.115529 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:28.117072 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:28.117654 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:28.117848 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:28.121124 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:28.121130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:28.121144 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:28.121178 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:28.123087 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:28.123614 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:28.123619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:28.123623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:28.123744 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:28.124117 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:28.124122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:28.124124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:28.124681 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:28.124823 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:28.125777 (+ 954us) server_negotiation.cc:299] Negotiation successful
0810 22:32:28.126103 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":124,"threads_started":1}
I20260810 22:32:28.126710 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:28.110873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:44687 (local address 127.1.221.131:43589)
0810 22:32:28.111341 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:28.111372 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:28.111422 (+ 50us) server_negotiation.cc:407] Connection header received
0810 22:32:28.112255 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:28.112262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:28.112641 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:28.112903 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:28.114457 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:28.115529 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:28.117693 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:28.118454 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:28.118627 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:28.122064 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:28.122069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:28.122106 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:28.122164 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:28.124320 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:28.124797 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:28.124802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:28.124806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:28.124918 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:28.125197 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:28.125200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:28.125201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:28.125766 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:28.125897 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:28.126254 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 22:32:28.126498 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":83,"threads_started":1}
W20260810 22:32:28.129823 20561 leader_election.cc:341] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130:36143): Not found: Tablet not found: 2a8fa7445b9b414bb589676e94245087
W20260810 22:32:28.130059 20561 leader_election.cc:341] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589): Not found: Tablet not found: 2a8fa7445b9b414bb589676e94245087
I20260810 22:32:28.130142 20561 leader_election.cc:302] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef20c5cdc4342d895474b7513b901e4; no voters: 86aae5efd275476fa0cd52dd3b0edb9d, 9b0673f9c4c14dc9b244fda13157a1dc
I20260810 22:32:28.130448 20997 raft_consensus.cc:2726] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:32:28.482563 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:28.482248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:49633)
0810 22:32:28.482384 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:28.482474 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 22:32:28.620831 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:28.620395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:38459)
0810 22:32:28.620674 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:28.620729 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
I20260810 22:32:28.813779 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:28.813270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:51227)
0810 22:32:28.813558 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:28.813680 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260810 22:32:29.483978 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:29.483408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:54063)
0810 22:32:29.483740 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:29.483829 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":82,"threads_started":1}
I20260810 22:32:29.622148 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:29.621695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:51557)
0810 22:32:29.621962 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:29.622025 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260810 22:32:29.814976 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:29.814509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:51045)
0810 22:32:29.814809 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:29.814865 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
I20260810 22:32:30.237809 21008 raft_consensus.cc:482] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:32:30.237953 21008 raft_consensus.cc:504] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:30.238340 21008 leader_election.cc:288] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589), 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130:36143)
W20260810 22:32:30.239382 20561 leader_election.cc:341] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130:36143): Not found: Tablet not found: 2a8fa7445b9b414bb589676e94245087
W20260810 22:32:30.239558 20561 leader_election.cc:341] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589): Not found: Tablet not found: 2a8fa7445b9b414bb589676e94245087
I20260810 22:32:30.239636 20561 leader_election.cc:302] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef20c5cdc4342d895474b7513b901e4; no voters: 86aae5efd275476fa0cd52dd3b0edb9d, 9b0673f9c4c14dc9b244fda13157a1dc
I20260810 22:32:30.239835 21008 raft_consensus.cc:2726] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:32:30.485306 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:30.484850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:46317)
0810 22:32:30.485135 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:30.485207 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260810 22:32:30.623332 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:30.622889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:34361)
0810 22:32:30.623155 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:30.623209 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
I20260810 22:32:30.816146 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:30.815741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:38645)
0810 22:32:30.816002 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:30.816060 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
I20260810 22:32:31.486754 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:31.486290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:43009)
0810 22:32:31.486596 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:31.486656 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":81,"threads_started":1}
I20260810 22:32:31.624563 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:31.624145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:55217)
0810 22:32:31.624396 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:31.624467 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260810 22:32:31.817466 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:31.817046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:40643)
0810 22:32:31.817314 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:31.817371 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260810 22:32:32.324555 21015 raft_consensus.cc:482] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:32:32.324702 21015 raft_consensus.cc:504] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:32.325217 21015 leader_election.cc:288] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589), 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130:36143)
W20260810 22:32:32.326135 20561 leader_election.cc:341] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130:36143): Not found: Tablet not found: 2a8fa7445b9b414bb589676e94245087
W20260810 22:32:32.326284 20561 leader_election.cc:341] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589): Not found: Tablet not found: 2a8fa7445b9b414bb589676e94245087
I20260810 22:32:32.326344 20561 leader_election.cc:302] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef20c5cdc4342d895474b7513b901e4; no voters: 86aae5efd275476fa0cd52dd3b0edb9d, 9b0673f9c4c14dc9b244fda13157a1dc
I20260810 22:32:32.326556 21015 raft_consensus.cc:2726] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:32:32.487973 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:32.487567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:60495)
0810 22:32:32.487822 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:32.487878 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260810 22:32:32.625869 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:32.625372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:44573)
0810 22:32:32.625715 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:32.625775 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":102,"threads_started":1}
I20260810 22:32:32.818843 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:32.818289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:60581)
0810 22:32:32.818618 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:32.818746 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":135,"threads_started":1}
I20260810 22:32:33.489276 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:33.488867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:60293)
0810 22:32:33.489121 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:33.489178 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260810 22:32:33.627142 21020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:33.626645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:59123)
0810 22:32:33.626978 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:33.627041 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":75,"threads_started":1}
I20260810 22:32:33.820047 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:33.819574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:35995)
0810 22:32:33.819893 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:33.819952 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":70,"threads_started":1}
I20260810 22:32:34.490633 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:34.490198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:50321)
0810 22:32:34.490471 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:34.490534 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260810 22:32:34.628319 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:34.627912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:58689)
0810 22:32:34.628168 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:34.628224 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260810 22:32:34.821408 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:34.820989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:46345)
0810 22:32:34.821253 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:34.821312 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260810 22:32:35.491930 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:35.491487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:39759)
0810 22:32:35.491774 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:35.491832 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":101,"threads_started":1}
I20260810 22:32:35.629715 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:35.629145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:54001)
0810 22:32:35.629496 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:35.629558 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":135,"threads_started":1}
I20260810 22:32:35.822642 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:35.822216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:43805)
0810 22:32:35.822487 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:35.822544 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260810 22:32:36.493242 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:36.492795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:42799)
0810 22:32:36.493071 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:36.493138 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
I20260810 22:32:36.630966 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:36.630507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:40453)
0810 22:32:36.630798 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:36.630865 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260810 22:32:36.823856 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:36.823420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:33727)
0810 22:32:36.823702 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:36.823762 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":76,"threads_started":1}
I20260810 22:32:37.161928 21033 raft_consensus.cc:482] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:32:37.162099 21033 raft_consensus.cc:504] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86aae5efd275476fa0cd52dd3b0edb9d" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43589 } } peers { permanent_uuid: "9b0673f9c4c14dc9b244fda13157a1dc" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 36143 } } peers { permanent_uuid: "9ef20c5cdc4342d895474b7513b901e4" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40933 } }
I20260810 22:32:37.162575 21033 leader_election.cc:288] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589), 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130:36143)
W20260810 22:32:37.163408 20561 leader_election.cc:341] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b0673f9c4c14dc9b244fda13157a1dc (127.1.221.130:36143): Not found: Tablet not found: 2a8fa7445b9b414bb589676e94245087
W20260810 22:32:37.163533 20561 leader_election.cc:341] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 86aae5efd275476fa0cd52dd3b0edb9d (127.1.221.131:43589): Not found: Tablet not found: 2a8fa7445b9b414bb589676e94245087
I20260810 22:32:37.163573 20561 leader_election.cc:302] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef20c5cdc4342d895474b7513b901e4; no voters: 86aae5efd275476fa0cd52dd3b0edb9d, 9b0673f9c4c14dc9b244fda13157a1dc
I20260810 22:32:37.163704 21033 raft_consensus.cc:2726] T 2a8fa7445b9b414bb589676e94245087 P 9ef20c5cdc4342d895474b7513b901e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:32:37.494499 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:37.494061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:50147)
0810 22:32:37.494345 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:37.494402 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
I20260810 22:32:37.632285 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:37.631833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:39889)
0810 22:32:37.632123 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:37.632186 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260810 22:32:37.825042 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:37.824611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:48603)
0810 22:32:37.824888 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:37.824947 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":77,"threads_started":1}
I20260810 22:32:38.495733 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:38.495317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:49871)
0810 22:32:38.495580 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:38.495637 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20260810 22:32:38.633714 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:38.633161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:55431)
0810 22:32:38.633471 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:38.633535 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":104,"threads_started":1}
I20260810 22:32:38.826625 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:38.826104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:60053)
0810 22:32:38.826456 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:38.826518 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":91,"threads_started":1}
I20260810 22:32:39.496999 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:39.496555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:58921)
0810 22:32:39.496841 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:39.496899 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":99,"threads_started":1}
I20260810 22:32:39.634961 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:39.634510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:42775)
0810 22:32:39.634796 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:39.634860 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":93,"threads_started":1}
I20260810 22:32:39.827801 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:39.827370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:57459)
0810 22:32:39.827647 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:39.827703 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":92,"threads_started":1}
I20260810 22:32:40.498241 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:40.497818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:39631)
0810 22:32:40.498089 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:40.498146 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260810 22:32:40.636240 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:40.635744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:38319)
0810 22:32:40.636028 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:40.636092 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":107,"threads_started":1}
I20260810 22:32:40.829039 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:40.828622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:52891)
0810 22:32:40.828882 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:40.828942 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260810 22:32:41.499611 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:41.499212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.129:49341)
0810 22:32:41.499459 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:41.499515 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
I20260810 22:32:41.637717 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:41.637216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.130:41515)
0810 22:32:41.637495 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:41.637557 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
I20260810 22:32:41.815058 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 20499
I20260810 22:32:41.825466 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 20689
I20260810 22:32:41.830518 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:41.830078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44603 (local address 127.1.221.131:36803)
0810 22:32:41.830366 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:41.830422 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.190:44603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
I20260810 22:32:41.833071 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 20840
2026-08-10T22:32:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59512 ms)
[----------] 30 tests from SecurityITest (248645 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:32:41 dist-test-slave-njxd krb5kdc[21053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:32:41 dist-test-slave-njxd krb5kdc[21053](info): set up 2 sockets
Aug 10 22:32:41 dist-test-slave-njxd krb5kdc[21053](info): commencing operation
krb5kdc: starting...
W20260810 22:32:42.941877 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:32:43 dist-test-slave-njxd krb5kdc[21053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:32:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:32:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:32:43.073133 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:41987
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:41987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:43.187589 21067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:43.187848 21067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:43.187886 21067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:43.191833 21067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:32:43.191890 21067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:43.191910 21067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:43.191927 21067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:32:43.191942 21067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:32:43.197077 21067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:41987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:41987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:43.198108 21067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:43.199054 21067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:43.207535 21072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:43.207546 21074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:43.207598 21067 server_base.cc:1034] running on GCE node
W20260810 22:32:43.207546 21076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:43.208328 21067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:43.209452 21067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:43.210635 21067 hybrid_clock.cc:648] HybridClock initialized: now 1786401163210617 us; error 36 us; skew 500 ppm
Aug 10 22:32:43 dist-test-slave-njxd krb5kdc[21053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401163, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:43.213368 21067 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:32:43.214283 21067 webserver.cc:460] Webserver started at http://127.1.221.190:38749/ using document root <none> and password file <none>
I20260810 22:32:43.214845 21067 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:43.214907 21067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:43.215075 21067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:43.217094 21067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "7ae6b99028e74cc1b6fafac5db0d71d7"
format_stamp: "Formatted at 2026-08-10 22:32:43 on dist-test-slave-njxd"
server_key: "0ceb70c68df1d358c4022fedbd9b09e8"
server_key_iv: "3c2d16e382bf1116d54c065e1fd30bec"
server_key_version: "encryptionkey@0"
I20260810 22:32:43.217718 21067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "7ae6b99028e74cc1b6fafac5db0d71d7"
format_stamp: "Formatted at 2026-08-10 22:32:43 on dist-test-slave-njxd"
server_key: "0ceb70c68df1d358c4022fedbd9b09e8"
server_key_iv: "3c2d16e382bf1116d54c065e1fd30bec"
server_key_version: "encryptionkey@0"
I20260810 22:32:43.221411 21067 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:32:43.223912 21084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:43.225229 21067 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:43.225359 21067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "7ae6b99028e74cc1b6fafac5db0d71d7"
format_stamp: "Formatted at 2026-08-10 22:32:43 on dist-test-slave-njxd"
server_key: "0ceb70c68df1d358c4022fedbd9b09e8"
server_key_iv: "3c2d16e382bf1116d54c065e1fd30bec"
server_key_version: "encryptionkey@0"
I20260810 22:32:43.225471 21067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:43.244638 21067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:43.247646 21067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:43.247874 21067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:43.255514 21067 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:41987
I20260810 22:32:43.255546 21144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:41987 every 8 connection(s)
I20260810 22:32:43.256480 21067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:32:43.259382 21145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:43.260921 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 21067
I20260810 22:32:43.261082 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:32:43.261371 1910 external_mini_cluster.cc:1428] Setting key 26c15aeca7dbf972ee2805c797b123c2
I20260810 22:32:43.265529 21145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7: Bootstrap starting.
I20260810 22:32:43.267871 21145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:43.268841 21145 log.cc:824] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:32:43 dist-test-slave-njxd krb5kdc[21053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:32:43.271078 21145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7: No bootstrap required, opened a new log
I20260810 22:32:43.274660 21145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae6b99028e74cc1b6fafac5db0d71d7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41987 } }
I20260810 22:32:43.274915 21145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:43.275027 21145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ae6b99028e74cc1b6fafac5db0d71d7, State: Initialized, Role: FOLLOWER
I20260810 22:32:43.275488 21145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae6b99028e74cc1b6fafac5db0d71d7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41987 } }
I20260810 22:32:43.275647 21145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:32:43.275748 21145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:32:43.275887 21145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:43.276364 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:43.262901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.1.221.190:41987)
0810 22:32:43.263343 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:43.263358 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:43.263393 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:32:43.264021 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:43.264038 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:43.264363 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:43.264697 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:43.265490 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:43.266593 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:43.267471 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:43.267950 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:43.268131 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:43.271001 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:43.271006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:43.271020 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:43.271056 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:43.272920 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:43.273434 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:43.273439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:43.273444 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:43.273562 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:43.273911 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:43.273914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:43.273916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:43.274322 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:43.274541 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:43.274871 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 22:32:43.275388 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":134,"threads_started":1}
I20260810 22:32:43.276911 21145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae6b99028e74cc1b6fafac5db0d71d7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41987 } }
I20260810 22:32:43.277251 21145 leader_election.cc:302] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ae6b99028e74cc1b6fafac5db0d71d7; no voters:
I20260810 22:32:43.277627 21145 leader_election.cc:288] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:32:43.277702 21150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:43.277945 21150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [term 1 LEADER]: Becoming Leader. State: Replica: 7ae6b99028e74cc1b6fafac5db0d71d7, State: Running, Role: LEADER
I20260810 22:32:43.278311 21150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae6b99028e74cc1b6fafac5db0d71d7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41987 } }
I20260810 22:32:43.278939 21145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:32:43.279500 21150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ae6b99028e74cc1b6fafac5db0d71d7. Latest consensus state: current_term: 1 leader_uuid: "7ae6b99028e74cc1b6fafac5db0d71d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae6b99028e74cc1b6fafac5db0d71d7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41987 } } }
I20260810 22:32:43.279704 21150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:43.280107 21151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ae6b99028e74cc1b6fafac5db0d71d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae6b99028e74cc1b6fafac5db0d71d7" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 41987 } } }
I20260810 22:32:43.280184 21157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:32:43.280303 21151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:43.284049 21157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:32:43.289881 21157 catalog_manager.cc:1347] Generated new cluster ID: a9332c39e4d34fdc9b7fe03c3a6097c8
I20260810 22:32:43.289991 21157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:32:43.300503 21157 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:32:43.301399 21157 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:32:43.316560 21157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7: Generated new TSK 0
I20260810 22:32:43.317318 21157 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:32:43.353284 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41987
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:43.469666 21174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:43.469949 21174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:43.469993 21174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:43.474757 21174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:43.474845 21174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:43.474907 21174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:32:43.480643 21174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:43.481949 21174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:43.482969 21174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:43.492959 21183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:43.492990 21180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:43.492965 21181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:43.493108 21174 server_base.cc:1034] running on GCE node
I20260810 22:32:43.493654 21174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:43.494271 21174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:43.495445 21174 hybrid_clock.cc:648] HybridClock initialized: now 1786401163495420 us; error 35 us; skew 500 ppm
Aug 10 22:32:43 dist-test-slave-njxd krb5kdc[21053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401163, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:43.498283 21174 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:32:43.499176 21174 webserver.cc:460] Webserver started at http://127.1.221.129:36211/ using document root <none> and password file <none>
I20260810 22:32:43.499866 21174 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:43.499958 21174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:43.500207 21174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:43.502368 21174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "dca650ce91bd4b3d8e70576e6d2edd06"
format_stamp: "Formatted at 2026-08-10 22:32:43 on dist-test-slave-njxd"
server_key: "487952a51b1f2159ad71210107455f96"
server_key_iv: "a0375a713a42d5e5a2157ac5dd271110"
server_key_version: "encryptionkey@0"
I20260810 22:32:43.502950 21174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "dca650ce91bd4b3d8e70576e6d2edd06"
format_stamp: "Formatted at 2026-08-10 22:32:43 on dist-test-slave-njxd"
server_key: "487952a51b1f2159ad71210107455f96"
server_key_iv: "a0375a713a42d5e5a2157ac5dd271110"
server_key_version: "encryptionkey@0"
I20260810 22:32:43.506846 21174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:32:43.509344 21192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:43.510702 21174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:32:43.510857 21174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "dca650ce91bd4b3d8e70576e6d2edd06"
format_stamp: "Formatted at 2026-08-10 22:32:43 on dist-test-slave-njxd"
server_key: "487952a51b1f2159ad71210107455f96"
server_key_iv: "a0375a713a42d5e5a2157ac5dd271110"
server_key_version: "encryptionkey@0"
I20260810 22:32:43.510975 21174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:43.532662 21174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:43.535904 21174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:43.536139 21174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:43.536820 21174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:43.537725 21174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:43.537799 21174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:43.537873 21174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:43.537918 21174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:43.547133 21174 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:45061
I20260810 22:32:43.547164 21322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:45061 every 8 connection(s)
I20260810 22:32:43.548076 21174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:32:43.550593 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 21174
I20260810 22:32:43.550742 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:32:43.551090 1910 external_mini_cluster.cc:1428] Setting key 6253788f31350b73875b0b2b2d6f75bc
Aug 10 22:32:43 dist-test-slave-njxd krb5kdc[21053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401163, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260810 22:32:43.563380 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:43.550024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:40259 (local address 127.1.221.190:41987)
0810 22:32:43.550182 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:43.550186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:43.551203 (+ 1017us) server_negotiation.cc:407] Connection header received
0810 22:32:43.552637 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:43.552640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:43.552697 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:43.552784 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:43.554173 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:43.554718 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:43.555979 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:43.556393 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:43.556519 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:43.559284 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:43.559291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:43.559297 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:43.559337 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:43.561123 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:43.561693 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:43.561698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:43.561700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:43.561799 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:43.562199 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:43.562205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:43.562210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:43.562466 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:43.562572 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:43.563143 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 22:32:43.563253 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:32:43.564239 21325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:43.550457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41987 (local address 127.1.221.129:40259)
0810 22:32:43.551034 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:43.551083 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:43.552397 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:43.552967 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:43.552974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:43.553370 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:43.553980 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:43.553996 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:43.554862 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:43.554873 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:43.555838 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:43.555848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:43.556573 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:43.556577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:43.556742 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:43.557408 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:43.557425 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:43.559095 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:43.561273 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:43.561278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:43.561292 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:43.561539 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:43.561900 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:43.561902 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:43.561903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:43.562037 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:43.562573 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:43.562578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:43.562862 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 22:32:43.563034 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 22:32:43.563316 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":178,"threads_started":1}
I20260810 22:32:43.565614 21323 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260810 22:32:43.565923 21323 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:43.566440 21323 heartbeater.cc:511] Master 127.1.221.190:41987 requested a full tablet report, sending...
I20260810 22:32:43.568176 21104 ts_manager.cc:194] Registered new tserver with Master: dca650ce91bd4b3d8e70576e6d2edd06 (127.1.221.129:45061)
I20260810 22:32:43.569301 21104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:40259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:32:43.584353 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41987
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:43.699796 21328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:43.700116 21328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:43.700213 21328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:43.704169 21328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:43.704257 21328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:43.704345 21328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:32:43.709568 21328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:43.710712 21328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:43.711799 21328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:43.721596 21333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:43.721621 21334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:43.721782 21328 server_base.cc:1034] running on GCE node
W20260810 22:32:43.721596 21336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:43.722220 21328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:43.722815 21328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:43.723978 21328 hybrid_clock.cc:648] HybridClock initialized: now 1786401163723960 us; error 30 us; skew 500 ppm
Aug 10 22:32:43 dist-test-slave-njxd krb5kdc[21053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401163, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:43.726825 21328 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:32:43.727694 21328 webserver.cc:460] Webserver started at http://127.1.221.130:41981/ using document root <none> and password file <none>
I20260810 22:32:43.728272 21328 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:43.728365 21328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:43.728605 21328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:43.730887 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "5208151929284e429830ef17414b7676"
format_stamp: "Formatted at 2026-08-10 22:32:43 on dist-test-slave-njxd"
server_key: "54cd65d36869e4a3f8d81335e1a44869"
server_key_iv: "42b78ef48146c9496435e44b69e526a3"
server_key_version: "encryptionkey@0"
I20260810 22:32:43.731487 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "5208151929284e429830ef17414b7676"
format_stamp: "Formatted at 2026-08-10 22:32:43 on dist-test-slave-njxd"
server_key: "54cd65d36869e4a3f8d81335e1a44869"
server_key_iv: "42b78ef48146c9496435e44b69e526a3"
server_key_version: "encryptionkey@0"
I20260810 22:32:43.735284 21328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:32:43.737720 21343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:43.738919 21328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:32:43.739063 21328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "5208151929284e429830ef17414b7676"
format_stamp: "Formatted at 2026-08-10 22:32:43 on dist-test-slave-njxd"
server_key: "54cd65d36869e4a3f8d81335e1a44869"
server_key_iv: "42b78ef48146c9496435e44b69e526a3"
server_key_version: "encryptionkey@0"
I20260810 22:32:43.739179 21328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:43.754338 21328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:43.757452 21328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:43.757707 21328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:43.758327 21328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:43.759163 21328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:43.759245 21328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:43.759320 21328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:43.759375 21328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:43.768362 21328 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:38355
I20260810 22:32:43.768400 21466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:38355 every 8 connection(s)
I20260810 22:32:43.769222 21328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:32:43.772010 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 21328
I20260810 22:32:43.772157 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:32:43.772528 1910 external_mini_cluster.cc:1428] Setting key 7ee74ff94243ce89d2f2391fcb8e6243
Aug 10 22:32:43 dist-test-slave-njxd krb5kdc[21053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401163, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260810 22:32:43.784592 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:43.771036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:52637 (local address 127.1.221.190:41987)
0810 22:32:43.771172 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:43.771176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:43.772182 (+ 1006us) server_negotiation.cc:407] Connection header received
0810 22:32:43.773240 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:43.773244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:43.773298 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:43.773378 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:43.774872 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:43.775531 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:43.777004 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:43.777230 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:43.777341 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:43.780222 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:43.780229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:43.780235 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:43.780272 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:43.782192 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:43.782805 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:43.782810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:43.782812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:43.782911 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:43.783332 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:43.783336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:43.783338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:43.783601 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:43.783721 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:43.784260 (+ 539us) server_negotiation.cc:299] Negotiation successful
0810 22:32:43.784399 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:32:43.785619 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:43.771416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41987 (local address 127.1.221.130:52637)
0810 22:32:43.771930 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:43.771991 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:43.773030 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:43.773539 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:43.773545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:43.774034 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:43.774715 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:43.774738 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:43.775673 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:43.775680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:43.776874 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:43.776883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:43.777384 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:43.777388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:43.777556 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:43.778238 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:43.778254 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:43.780012 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:43.782349 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:43.782355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:43.782369 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:43.782698 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:43.783034 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:43.783037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:43.783038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:43.783213 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:43.783720 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:43.783725 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:43.783981 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 22:32:43.784150 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 22:32:43.784578 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260810 22:32:43.786885 21467 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260810 22:32:43.787194 21467 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:43.787778 21467 heartbeater.cc:511] Master 127.1.221.190:41987 requested a full tablet report, sending...
I20260810 22:32:43.788870 21104 ts_manager.cc:194] Registered new tserver with Master: 5208151929284e429830ef17414b7676 (127.1.221.130:38355)
I20260810 22:32:43.789744 21104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:52637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:32:43.804499 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41987
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:35293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:43.919452 21474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:43.919701 21474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:43.919771 21474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:43.923790 21474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:43.923846 21474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:43.923902 21474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:32:43.929344 21474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:35293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:41987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:43.930411 21474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:43.931447 21474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:43.941481 21480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:43.941480 21479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:43.941494 21482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:43.941756 21474 server_base.cc:1034] running on GCE node
I20260810 22:32:43.942328 21474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:43.942904 21474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:43.944100 21474 hybrid_clock.cc:648] HybridClock initialized: now 1786401163944059 us; error 53 us; skew 500 ppm
Aug 10 22:32:43 dist-test-slave-njxd krb5kdc[21053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401163, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:43.946939 21474 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:32:43.948101 21474 webserver.cc:460] Webserver started at http://127.1.221.131:42653/ using document root <none> and password file <none>
I20260810 22:32:43.948849 21474 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:43.948941 21474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:43.949184 21474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:43.951385 21474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "70a6c717d76e4f7b96fe61ee8a42674e"
format_stamp: "Formatted at 2026-08-10 22:32:43 on dist-test-slave-njxd"
server_key: "d88c62a0579681ad293466285c48c1fd"
server_key_iv: "f78b71d8b87270e969e20f75aa400cad"
server_key_version: "encryptionkey@0"
I20260810 22:32:43.952138 21474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "70a6c717d76e4f7b96fe61ee8a42674e"
format_stamp: "Formatted at 2026-08-10 22:32:43 on dist-test-slave-njxd"
server_key: "d88c62a0579681ad293466285c48c1fd"
server_key_iv: "f78b71d8b87270e969e20f75aa400cad"
server_key_version: "encryptionkey@0"
I20260810 22:32:43.956107 21474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:32:43.958772 21490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:43.960155 21474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:32:43.960302 21474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "70a6c717d76e4f7b96fe61ee8a42674e"
format_stamp: "Formatted at 2026-08-10 22:32:43 on dist-test-slave-njxd"
server_key: "d88c62a0579681ad293466285c48c1fd"
server_key_iv: "f78b71d8b87270e969e20f75aa400cad"
server_key_version: "encryptionkey@0"
I20260810 22:32:43.960415 21474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:43.998605 21474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:44.002197 21474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:44.002417 21474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:44.003073 21474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:44.003947 21474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:44.003995 21474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:44.004065 21474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:44.004103 21474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:44.013891 21474 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:35321
I20260810 22:32:44.013999 21614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:35321 every 8 connection(s)
I20260810 22:32:44.014948 21474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:32:44.022239 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 21474
I20260810 22:32:44.022383 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:32:44.022706 1910 external_mini_cluster.cc:1428] Setting key f2a6488a7dbcab87031e4c027662ebd7
Aug 10 22:32:44 dist-test-slave-njxd krb5kdc[21053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401163, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:32:44.030208 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.016924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:41097 (local address 127.1.221.190:41987)
0810 22:32:44.017067 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:44.017071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:44.018057 (+ 986us) server_negotiation.cc:407] Connection header received
0810 22:32:44.019454 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:44.019457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:44.019523 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:44.019638 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:44.020999 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.021533 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.023181 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.023424 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.023530 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.026402 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:44.026406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:44.026408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:44.026433 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:44.027979 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:44.028516 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:44.028520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:44.028522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:44.028617 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:44.028990 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:44.028993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:44.028995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:44.029228 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:44.029336 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:44.029912 (+ 576us) server_negotiation.cc:299] Negotiation successful
0810 22:32:44.030043 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:32:44.031100 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.017254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:41987 (local address 127.1.221.131:41097)
0810 22:32:44.017856 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:44.017935 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:44.019217 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:44.019830 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:44.019837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:44.020306 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:44.020836 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.020851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.021698 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.021709 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.023036 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.023059 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.023606 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.023614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.023810 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.024494 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:44.024511 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:44.026227 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:44.028113 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:44.028119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:44.028135 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:44.028408 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:44.028731 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:44.028736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:44.028738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:44.028890 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:44.029336 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:44.029344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:44.029662 (+ 318us) client_negotiation.cc:769] Sending connection context
0810 22:32:44.029869 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 22:32:44.030242 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":127,"threads_started":1}
I20260810 22:32:44.032219 21615 heartbeater.cc:347] Connected to a master server at 127.1.221.190:41987
I20260810 22:32:44.032442 21615 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:44.032887 21615 heartbeater.cc:511] Master 127.1.221.190:41987 requested a full tablet report, sending...
I20260810 22:32:44.033964 21104 ts_manager.cc:194] Registered new tserver with Master: 70a6c717d76e4f7b96fe61ee8a42674e (127.1.221.131:35321)
I20260810 22:32:44.034483 21104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:41097
I20260810 22:32:44.036854 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:32:44.046677 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.039139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.1.221.190:41987)
0810 22:32:44.039264 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:44.039268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:44.039419 (+ 151us) server_negotiation.cc:407] Connection header received
0810 22:32:44.039508 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:44.039511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:44.039548 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:44.039638 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:44.040286 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.040789 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.041782 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.042111 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.042204 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.043674 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:44.043677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:44.043679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:44.043701 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:44.045194 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:44.045562 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:44.045564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:44.045565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:44.045703 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:44.045939 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:44.045941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:44.045942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:44.046168 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:44.046270 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:44.046445 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 22:32:44.046535 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:32:44.049717 21104 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:32:44.052893 21104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:32:44.067853 21628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.061934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45061 (local address 127.0.0.1:48602)
0810 22:32:44.063119 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:44.063134 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:44.063276 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:44.063743 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:44.063746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:44.063758 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:44.063921 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.063930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.065227 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.065233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.066439 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.066449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.067278 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.067282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.067418 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.067455 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 22:32:44.067564 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 22:32:44.067698 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"mutex_wait_us":39,"thread_start_us":104,"threads_started":1}
I20260810 22:32:44.067858 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.062102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.1.221.129:45061)
0810 22:32:44.062971 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:44.062990 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:44.063258 (+ 268us) server_negotiation.cc:407] Connection header received
0810 22:32:44.063390 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:44.063398 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:44.063595 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:44.063757 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:44.064039 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.065106 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.066568 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.067141 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.067279 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.067452 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:44.067615 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 22:32:44.067710 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":119,"threads_started":1}
I20260810 22:32:44.068994 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.062643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:35321 (local address 127.0.0.1:43046)
0810 22:32:44.063477 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:44.063489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:44.063559 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:44.064287 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:44.064290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:44.064308 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:44.064497 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.064506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.065887 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.065890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.067273 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.067285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.068581 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.068585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.068679 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.068710 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:32:44.068785 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 22:32:44.068850 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":121,"thread_start_us":100,"threads_started":1}
I20260810 22:32:44.069051 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.063053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.1.221.131:35321)
0810 22:32:44.063815 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:44.063826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:44.063846 (+ 20us) server_negotiation.cc:407] Connection header received
0810 22:32:44.063921 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:44.063926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:44.064151 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:44.064308 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:44.064635 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.065753 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.067432 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.068274 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.068378 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.068512 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:44.068818 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 22:32:44.068915 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":133,"threads_started":1}
I20260810 22:32:44.069346 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.061898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.1.221.130:38355)
0810 22:32:44.062727 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:44.062741 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:44.062771 (+ 30us) server_negotiation.cc:407] Connection header received
0810 22:32:44.063036 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:44.063043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:44.063292 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:44.063427 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:44.063984 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.065543 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.067432 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.068274 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.068378 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.068669 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:44.069064 (+ 395us) server_negotiation.cc:299] Negotiation successful
0810 22:32:44.069195 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":158,"threads_started":1}
I20260810 22:32:44.069679 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.061744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:38355 (local address 127.0.0.1:33236)
0810 22:32:44.062652 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:44.062706 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:44.062840 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:44.063630 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:44.063635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:44.063654 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:44.063879 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.063888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.065858 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.065864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.067280 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.067286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.068725 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.068729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.068819 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.068837 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 22:32:44.068896 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 22:32:44.068948 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":185,"thread_start_us":117,"threads_started":1}
I20260810 22:32:44.071630 21246 tablet_service.cc:1467] Processing CreateTablet for tablet 76a5769875d04d30b97d0ca96c6ceb87 (DEFAULT_TABLE table=test-table [id=d65a4774c41b407faa5c70a574e76b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:44.071976 21542 tablet_service.cc:1467] Processing CreateTablet for tablet 76a5769875d04d30b97d0ca96c6ceb87 (DEFAULT_TABLE table=test-table [id=d65a4774c41b407faa5c70a574e76b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:44.072691 21246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a5769875d04d30b97d0ca96c6ceb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:44.072997 21542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a5769875d04d30b97d0ca96c6ceb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:44.079751 21633 tablet_bootstrap.cc:492] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06: Bootstrap starting.
I20260810 22:32:44.079873 21396 tablet_service.cc:1467] Processing CreateTablet for tablet 76a5769875d04d30b97d0ca96c6ceb87 (DEFAULT_TABLE table=test-table [id=d65a4774c41b407faa5c70a574e76b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:44.080927 21396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a5769875d04d30b97d0ca96c6ceb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:44.081810 21634 tablet_bootstrap.cc:492] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e: Bootstrap starting.
I20260810 22:32:44.081951 21633 tablet_bootstrap.cc:654] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:44.082866 21633 log.cc:824] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:44.083981 21634 tablet_bootstrap.cc:654] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:44.085088 21633 tablet_bootstrap.cc:492] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06: No bootstrap required, opened a new log
I20260810 22:32:44.085295 21633 ts_tablet_manager.cc:1397] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:32:44.085397 21634 log.cc:824] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:44.087414 21637 tablet_bootstrap.cc:492] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676: Bootstrap starting.
I20260810 22:32:44.087486 21634 tablet_bootstrap.cc:492] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e: No bootstrap required, opened a new log
I20260810 22:32:44.087734 21634 ts_tablet_manager.cc:1397] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 22:32:44.089668 21637 tablet_bootstrap.cc:654] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:44.089802 21633 raft_consensus.cc:348] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5208151929284e429830ef17414b7676" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38355 } } peers { permanent_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45061 } } peers { permanent_uuid: "70a6c717d76e4f7b96fe61ee8a42674e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35321 } }
I20260810 22:32:44.090070 21633 raft_consensus.cc:374] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:44.090375 21633 raft_consensus.cc:729] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca650ce91bd4b3d8e70576e6d2edd06, State: Initialized, Role: FOLLOWER
I20260810 22:32:44.090438 21637 log.cc:824] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:44.090889 21633 consensus_queue.cc:260] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5208151929284e429830ef17414b7676" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38355 } } peers { permanent_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45061 } } peers { permanent_uuid: "70a6c717d76e4f7b96fe61ee8a42674e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35321 } }
I20260810 22:32:44.091748 21634 raft_consensus.cc:348] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5208151929284e429830ef17414b7676" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38355 } } peers { permanent_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45061 } } peers { permanent_uuid: "70a6c717d76e4f7b96fe61ee8a42674e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35321 } }
I20260810 22:32:44.091995 21633 ts_tablet_manager.cc:1428] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 22:32:44.091972 21323 heartbeater.cc:503] Master 127.1.221.190:41987 was elected leader, sending a full tablet report...
I20260810 22:32:44.092109 21634 raft_consensus.cc:374] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:44.092165 21634 raft_consensus.cc:729] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a6c717d76e4f7b96fe61ee8a42674e, State: Initialized, Role: FOLLOWER
I20260810 22:32:44.092595 21634 consensus_queue.cc:260] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5208151929284e429830ef17414b7676" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38355 } } peers { permanent_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45061 } } peers { permanent_uuid: "70a6c717d76e4f7b96fe61ee8a42674e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35321 } }
I20260810 22:32:44.092756 21637 tablet_bootstrap.cc:492] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676: No bootstrap required, opened a new log
I20260810 22:32:44.092993 21637 ts_tablet_manager.cc:1397] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260810 22:32:44.093258 21615 heartbeater.cc:503] Master 127.1.221.190:41987 was elected leader, sending a full tablet report...
I20260810 22:32:44.093529 21634 ts_tablet_manager.cc:1428] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:32:44.096539 21637 raft_consensus.cc:348] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5208151929284e429830ef17414b7676" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38355 } } peers { permanent_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45061 } } peers { permanent_uuid: "70a6c717d76e4f7b96fe61ee8a42674e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35321 } }
I20260810 22:32:44.096828 21637 raft_consensus.cc:374] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:44.096905 21637 raft_consensus.cc:729] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5208151929284e429830ef17414b7676, State: Initialized, Role: FOLLOWER
I20260810 22:32:44.097295 21637 consensus_queue.cc:260] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5208151929284e429830ef17414b7676" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38355 } } peers { permanent_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45061 } } peers { permanent_uuid: "70a6c717d76e4f7b96fe61ee8a42674e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35321 } }
I20260810 22:32:44.097892 21467 heartbeater.cc:503] Master 127.1.221.190:41987 was elected leader, sending a full tablet report...
I20260810 22:32:44.098184 21637 ts_tablet_manager.cc:1428] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 22:32:44.167383 21639 raft_consensus.cc:482] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:32:44.167609 21639 raft_consensus.cc:504] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5208151929284e429830ef17414b7676" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38355 } } peers { permanent_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45061 } } peers { permanent_uuid: "70a6c717d76e4f7b96fe61ee8a42674e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35321 } }
I20260810 22:32:44.168753 21639 leader_election.cc:288] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5208151929284e429830ef17414b7676 (127.1.221.130:38355), 70a6c717d76e4f7b96fe61ee8a42674e (127.1.221.131:35321)
I20260810 22:32:44.172775 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.169158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:38111 (local address 127.1.221.131:35321)
0810 22:32:44.169297 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:44.169302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:44.169556 (+ 254us) server_negotiation.cc:407] Connection header received
0810 22:32:44.169766 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:44.169771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:44.169820 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:44.169903 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:44.170182 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.170711 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.171793 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.172211 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.172341 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.172397 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:44.172567 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 22:32:44.172643 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:32:44.172803 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.169039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:35321 (local address 127.1.221.129:38111)
0810 22:32:44.169453 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:44.169466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:44.169639 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:44.169917 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:44.169920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:44.169937 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:44.170064 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.170070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.170825 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.170829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.171667 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.171676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.172334 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.172337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.172433 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.172458 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 22:32:44.172549 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 22:32:44.172619 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":182,"threads_started":1}
I20260810 22:32:44.172842 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.169160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:44649 (local address 127.1.221.130:38355)
0810 22:32:44.169299 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:44.169304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:44.169723 (+ 419us) server_negotiation.cc:407] Connection header received
0810 22:32:44.169872 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:44.169875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:44.169926 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:44.170012 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:44.170324 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.170812 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.171860 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.172271 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.172400 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.172455 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:44.172650 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 22:32:44.172721 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:32:44.172861 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.169051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:38355 (local address 127.1.221.129:44649)
0810 22:32:44.169585 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:44.169627 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:44.169754 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:44.170024 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:44.170026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:44.170043 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:44.170183 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.170189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.170932 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.170937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.171728 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.171736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.172396 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.172399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.172508 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.172537 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 22:32:44.172633 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 22:32:44.172704 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":153,"threads_started":1}
I20260810 22:32:44.173475 21418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a5769875d04d30b97d0ca96c6ceb87" candidate_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5208151929284e429830ef17414b7676" is_pre_election: true
I20260810 22:32:44.173662 21562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a5769875d04d30b97d0ca96c6ceb87" candidate_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70a6c717d76e4f7b96fe61ee8a42674e" is_pre_election: true
I20260810 22:32:44.173831 21418 raft_consensus.cc:2454] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca650ce91bd4b3d8e70576e6d2edd06 in term 0.
I20260810 22:32:44.173986 21562 raft_consensus.cc:2454] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca650ce91bd4b3d8e70576e6d2edd06 in term 0.
I20260810 22:32:44.174366 21195 leader_election.cc:302] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5208151929284e429830ef17414b7676, dca650ce91bd4b3d8e70576e6d2edd06; no voters:
I20260810 22:32:44.174690 21639 raft_consensus.cc:2781] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:32:44.174773 21639 raft_consensus.cc:482] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:32:44.174804 21639 raft_consensus.cc:3037] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:44.175916 21639 raft_consensus.cc:504] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5208151929284e429830ef17414b7676" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38355 } } peers { permanent_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45061 } } peers { permanent_uuid: "70a6c717d76e4f7b96fe61ee8a42674e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35321 } }
I20260810 22:32:44.176326 21639 leader_election.cc:288] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [CANDIDATE]: Term 1 election: Requested vote from peers 5208151929284e429830ef17414b7676 (127.1.221.130:38355), 70a6c717d76e4f7b96fe61ee8a42674e (127.1.221.131:35321)
I20260810 22:32:44.176671 21418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a5769875d04d30b97d0ca96c6ceb87" candidate_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5208151929284e429830ef17414b7676"
I20260810 22:32:44.176746 21562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a5769875d04d30b97d0ca96c6ceb87" candidate_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70a6c717d76e4f7b96fe61ee8a42674e"
I20260810 22:32:44.176815 21418 raft_consensus.cc:3037] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:44.176882 21562 raft_consensus.cc:3037] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:44.177983 21418 raft_consensus.cc:2454] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca650ce91bd4b3d8e70576e6d2edd06 in term 1.
I20260810 22:32:44.178037 21562 raft_consensus.cc:2454] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca650ce91bd4b3d8e70576e6d2edd06 in term 1.
I20260810 22:32:44.178300 21195 leader_election.cc:302] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5208151929284e429830ef17414b7676, dca650ce91bd4b3d8e70576e6d2edd06; no voters:
I20260810 22:32:44.178507 21639 raft_consensus.cc:2781] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:44.178767 21639 raft_consensus.cc:686] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [term 1 LEADER]: Becoming Leader. State: Replica: dca650ce91bd4b3d8e70576e6d2edd06, State: Running, Role: LEADER
I20260810 22:32:44.179109 21639 consensus_queue.cc:237] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5208151929284e429830ef17414b7676" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38355 } } peers { permanent_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45061 } } peers { permanent_uuid: "70a6c717d76e4f7b96fe61ee8a42674e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35321 } }
I20260810 22:32:44.182492 21103 catalog_manager.cc:5495] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca650ce91bd4b3d8e70576e6d2edd06 (127.1.221.129). New cstate: current_term: 1 leader_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5208151929284e429830ef17414b7676" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca650ce91bd4b3d8e70576e6d2edd06" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 45061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70a6c717d76e4f7b96fe61ee8a42674e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35321 } health_report { overall_health: UNKNOWN } } }
I20260810 22:32:44.222291 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.218556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48608 (local address 127.1.221.129:45061)
0810 22:32:44.218725 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:44.218729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:44.218744 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:32:44.218922 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:44.218925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:44.218984 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:44.219102 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:32:44.219388 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.219921 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.220790 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.221015 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.221131 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.221453 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:32:44.221902 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:32:44.222041 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:44.222082 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 22:32:44.222143 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 22:32:44.228389 21418 raft_consensus.cc:1261] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676 [term 1 FOLLOWER]: Refusing update from remote peer dca650ce91bd4b3d8e70576e6d2edd06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:44.228384 21562 raft_consensus.cc:1261] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e [term 1 FOLLOWER]: Refusing update from remote peer dca650ce91bd4b3d8e70576e6d2edd06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:44.229175 21645 consensus_queue.cc:1035] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5208151929284e429830ef17414b7676" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 38355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:32:44.229377 21639 consensus_queue.cc:1035] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70a6c717d76e4f7b96fe61ee8a42674e" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 35321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:32:44.238101 21649 mvcc.cc:205] Tried to move back new op lower bound from 7317099168672141312 to 7317099168480624640. Current Snapshot: MvccSnapshot[applied={T|T < 7317099168672141312}]
I20260810 22:32:44.249191 21103 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:44.249390 21103 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:44.252065 21103 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ae6b99028e74cc1b6fafac5db0d71d7: Sending DeleteTablet for 3 replicas of tablet 76a5769875d04d30b97d0ca96c6ceb87
I20260810 22:32:44.252861 21396 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a5769875d04d30b97d0ca96c6ceb87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:44 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:33236
I20260810 22:32:44.252889 21246 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a5769875d04d30b97d0ca96c6ceb87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:44 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:48602
I20260810 22:32:44.252920 21542 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a5769875d04d30b97d0ca96c6ceb87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:44 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:43046
I20260810 22:32:44.253391 21659 tablet_replica.cc:330] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06: stopping tablet replica
I20260810 22:32:44.253590 21661 tablet_replica.cc:330] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676: stopping tablet replica
I20260810 22:32:44.253798 21659 raft_consensus.cc:2229] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:32:44.254240 21658 tablet_replica.cc:330] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e: stopping tablet replica
I20260810 22:32:44.254263 21659 raft_consensus.cc:2258] T 76a5769875d04d30b97d0ca96c6ceb87 P dca650ce91bd4b3d8e70576e6d2edd06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:44.254489 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 21174
I20260810 22:32:44.254566 21658 raft_consensus.cc:2229] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:32:44.255640 21661 raft_consensus.cc:2229] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:32:44.255990 21661 raft_consensus.cc:2258] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:44.254884 21658 raft_consensus.cc:2258] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:44.258756 21658 ts_tablet_manager.cc:1905] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:44.260497 21661 ts_tablet_manager.cc:1905] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:44.262106 21658 ts_tablet_manager.cc:1918] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:44.262228 21658 log.cc:1196] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/76a5769875d04d30b97d0ca96c6ceb87
I20260810 22:32:44.262545 21658 ts_tablet_manager.cc:1939] T 76a5769875d04d30b97d0ca96c6ceb87 P 70a6c717d76e4f7b96fe61ee8a42674e: Deleting consensus metadata
I20260810 22:32:44.263154 21661 ts_tablet_manager.cc:1918] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:44.263243 21661 log.cc:1196] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/76a5769875d04d30b97d0ca96c6ceb87
I20260810 22:32:44.263556 21661 ts_tablet_manager.cc:1939] T 76a5769875d04d30b97d0ca96c6ceb87 P 5208151929284e429830ef17414b7676: Deleting consensus metadata
I20260810 22:32:44.263726 21088 catalog_manager.cc:4841] TS 70a6c717d76e4f7b96fe61ee8a42674e (127.1.221.131:35321): tablet 76a5769875d04d30b97d0ca96c6ceb87 (table test-table [id=d65a4774c41b407faa5c70a574e76b5c]) successfully deleted
I20260810 22:32:44.264667 21087 catalog_manager.cc:4841] TS 5208151929284e429830ef17414b7676 (127.1.221.130:38355): tablet 76a5769875d04d30b97d0ca96c6ceb87 (table test-table [id=d65a4774c41b407faa5c70a574e76b5c]) successfully deleted
W20260810 22:32:44.272197 21089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:45061 (error 108)
I20260810 22:32:44.272490 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 21328
I20260810 22:32:44.276182 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.275856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:45061 (local address 127.0.0.1:48624)
0810 22:32:44.276001 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:44.276078 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:45061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 22:32:44.276376 21089 catalog_manager.cc:4584] TS dca650ce91bd4b3d8e70576e6d2edd06 (127.1.221.129:45061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76a5769875d04d30b97d0ca96c6ceb87: Network error: Client connection negotiation failed: client connection to 127.1.221.129:45061: connect: Connection refused (error 111)
I20260810 22:32:44.281719 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 21474
I20260810 22:32:44.288996 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 21067
2026-08-10T22:32:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2455 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T22:32:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:32:44Z Disabled control of system clock
I20260810 22:32:44.323292 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43389
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:43389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 22:32:44.435386 21666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:44.435649 21666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:44.435686 21666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:44.439878 21666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:32:44.439940 21666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:44.439957 21666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:44.439975 21666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:32:44.440016 21666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:32:44.444830 21666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:43389
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:43389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:44.446053 21666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:44.447546 21666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:44.456607 21671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:44.456629 21672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:44.456733 21666 server_base.cc:1034] running on GCE node
W20260810 22:32:44.456601 21674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:44.457823 21666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:44.459031 21666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:44.460222 21666 hybrid_clock.cc:648] HybridClock initialized: now 1786401164460189 us; error 46 us; skew 500 ppm
I20260810 22:32:44.462306 21666 webserver.cc:460] Webserver started at http://127.1.221.190:35087/ using document root <none> and password file <none>
I20260810 22:32:44.462904 21666 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:44.463004 21666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:44.463250 21666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:44.465341 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "f2a96164a5e64a07b860562ac1783a1f"
format_stamp: "Formatted at 2026-08-10 22:32:44 on dist-test-slave-njxd"
server_key: "438d79a513d8f2080259580b328ed68d"
server_key_iv: "9149c24fc1e6266d7b6cff4ca572d4db"
server_key_version: "encryptionkey@0"
I20260810 22:32:44.465981 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "f2a96164a5e64a07b860562ac1783a1f"
format_stamp: "Formatted at 2026-08-10 22:32:44 on dist-test-slave-njxd"
server_key: "438d79a513d8f2080259580b328ed68d"
server_key_iv: "9149c24fc1e6266d7b6cff4ca572d4db"
server_key_version: "encryptionkey@0"
I20260810 22:32:44.469780 21666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:32:44.472287 21684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:44.473489 21666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:44.473654 21666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "f2a96164a5e64a07b860562ac1783a1f"
format_stamp: "Formatted at 2026-08-10 22:32:44 on dist-test-slave-njxd"
server_key: "438d79a513d8f2080259580b328ed68d"
server_key_iv: "9149c24fc1e6266d7b6cff4ca572d4db"
server_key_version: "encryptionkey@0"
I20260810 22:32:44.473769 21666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:44.491027 21666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:44.491747 21666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:44.491963 21666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:44.499068 21666 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:43389
I20260810 22:32:44.499094 21740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:43389 every 8 connection(s)
I20260810 22:32:44.500125 21666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:32:44.503191 21741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:44.508615 21741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f: Bootstrap starting.
I20260810 22:32:44.509826 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 21666
I20260810 22:32:44.509961 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:32:44.510249 1910 external_mini_cluster.cc:1428] Setting key 69a7538f39f2d8222873722118a4fca7
I20260810 22:32:44.511444 21741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:44.512400 21741 log.cc:824] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:44.514765 21741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f: No bootstrap required, opened a new log
I20260810 22:32:44.519001 21741 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a96164a5e64a07b860562ac1783a1f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43389 } }
I20260810 22:32:44.519279 21741 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:44.519369 21741 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a96164a5e64a07b860562ac1783a1f, State: Initialized, Role: FOLLOWER
I20260810 22:32:44.519837 21741 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a96164a5e64a07b860562ac1783a1f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43389 } }
I20260810 22:32:44.520009 21741 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:32:44.520092 21741 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:32:44.520190 21741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:44.520547 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.511441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.1.221.190:43389)
0810 22:32:44.512208 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:44.512230 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:44.512277 (+ 47us) server_negotiation.cc:407] Connection header received
0810 22:32:44.513078 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:44.513098 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:44.513529 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:44.513934 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:44.514391 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.515829 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.516697 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.517433 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.517618 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.518448 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:44.518455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:44.518471 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:44.518938 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:44.519005 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:44.519033 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:44.519145 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:44.519264 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 22:32:44.519533 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":162,"threads_started":1}
I20260810 22:32:44.521281 21741 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a96164a5e64a07b860562ac1783a1f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43389 } }
I20260810 22:32:44.521673 21741 leader_election.cc:302] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2a96164a5e64a07b860562ac1783a1f; no voters:
I20260810 22:32:44.522015 21741 leader_election.cc:288] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:32:44.522101 21746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:44.522279 21746 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [term 1 LEADER]: Becoming Leader. State: Replica: f2a96164a5e64a07b860562ac1783a1f, State: Running, Role: LEADER
I20260810 22:32:44.522650 21746 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a96164a5e64a07b860562ac1783a1f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43389 } }
I20260810 22:32:44.523327 21741 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:32:44.524660 21749 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2a96164a5e64a07b860562ac1783a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a96164a5e64a07b860562ac1783a1f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43389 } } }
I20260810 22:32:44.524785 21748 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2a96164a5e64a07b860562ac1783a1f. Latest consensus state: current_term: 1 leader_uuid: "f2a96164a5e64a07b860562ac1783a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a96164a5e64a07b860562ac1783a1f" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 43389 } } }
I20260810 22:32:44.524895 21748 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:44.524873 21749 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f [sys.catalog]: This master's current role is: LEADER
W20260810 22:32:44.528330 21762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:32:44.528416 21762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:32:44.528535 21758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:32:44.529249 21758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:32:44.534843 21758 catalog_manager.cc:1347] Generated new cluster ID: 84e6e0fca8cb4538aa1013ffa5875ae7
I20260810 22:32:44.534920 21758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:32:44.544006 21758 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:32:44.544870 21758 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:32:44.550040 21758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f: Generated new TSK 0
I20260810 22:32:44.550724 21758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:32:44.558511 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43389
--builtin_ntp_servers=127.1.221.148:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 22:32:44.673338 21766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:44.673686 21766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:44.673767 21766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:44.677917 21766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:44.677994 21766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:44.678081 21766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:32:44.682812 21766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:44.683939 21766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:44.685359 21766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:44.694952 21775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:44.694875 21773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:44.694868 21772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:44.695505 21766 server_base.cc:1034] running on GCE node
I20260810 22:32:44.695999 21766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:44.696602 21766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:44.697789 21766 hybrid_clock.cc:648] HybridClock initialized: now 1786401164697779 us; error 30 us; skew 500 ppm
I20260810 22:32:44.699692 21766 webserver.cc:460] Webserver started at http://127.1.221.129:34933/ using document root <none> and password file <none>
I20260810 22:32:44.700345 21766 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:44.700436 21766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:44.700690 21766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:44.702864 21766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "66aefc22b0dc490d87b3e29928d87c4b"
format_stamp: "Formatted at 2026-08-10 22:32:44 on dist-test-slave-njxd"
server_key: "ca9b3f78ead5381308b61f4821a4b3fb"
server_key_iv: "4ebd4ee8f50157f76f80b43f43989553"
server_key_version: "encryptionkey@0"
I20260810 22:32:44.703456 21766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "66aefc22b0dc490d87b3e29928d87c4b"
format_stamp: "Formatted at 2026-08-10 22:32:44 on dist-test-slave-njxd"
server_key: "ca9b3f78ead5381308b61f4821a4b3fb"
server_key_iv: "4ebd4ee8f50157f76f80b43f43989553"
server_key_version: "encryptionkey@0"
I20260810 22:32:44.707046 21766 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:32:44.709424 21781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:44.710747 21766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:44.710876 21766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "66aefc22b0dc490d87b3e29928d87c4b"
format_stamp: "Formatted at 2026-08-10 22:32:44 on dist-test-slave-njxd"
server_key: "ca9b3f78ead5381308b61f4821a4b3fb"
server_key_iv: "4ebd4ee8f50157f76f80b43f43989553"
server_key_version: "encryptionkey@0"
I20260810 22:32:44.710980 21766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:44.723343 21766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:44.724112 21766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:44.724296 21766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:44.724905 21766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:44.725983 21766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:44.726053 21766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:44.726132 21766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:44.726181 21766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:44.735131 21766 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:40163
I20260810 22:32:44.735178 21902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:40163 every 8 connection(s)
I20260810 22:32:44.736092 21766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:32:44.745661 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.737978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:46103 (local address 127.1.221.190:43389)
0810 22:32:44.738129 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:44.738133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:44.739260 (+ 1127us) server_negotiation.cc:407] Connection header received
0810 22:32:44.740712 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:44.740715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:44.740775 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:44.740851 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:44.741870 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.742431 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.743814 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.744056 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.744167 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.745204 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:44.745207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:44.745209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:44.745221 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:44.745243 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:44.745251 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:44.745312 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:44.745457 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 22:32:44.745513 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 22:32:44.746042 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 21766
I20260810 22:32:44.746176 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:32:44.746498 1910 external_mini_cluster.cc:1428] Setting key e0b11552c0ff1239229c35620b8e99d1
I20260810 22:32:44.746776 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.738411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43389 (local address 127.1.221.129:46103)
0810 22:32:44.739100 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:44.739142 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:44.740486 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:44.741020 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:44.741027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:44.741138 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:44.741725 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.741742 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.742567 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.742570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.743666 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.743677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.744227 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.744229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.744373 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.744938 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:32:44.744940 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:44.744979 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:32:44.745042 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:44.745328 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:44.745335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:44.745339 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 22:32:44.745433 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 22:32:44.745708 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
I20260810 22:32:44.748091 21903 heartbeater.cc:347] Connected to a master server at 127.1.221.190:43389
I20260810 22:32:44.748343 21903 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:44.748602 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43389
--builtin_ntp_servers=127.1.221.148:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 22:32:44.748916 21903 heartbeater.cc:511] Master 127.1.221.190:43389 requested a full tablet report, sending...
I20260810 22:32:44.750582 21703 ts_manager.cc:194] Registered new tserver with Master: 66aefc22b0dc490d87b3e29928d87c4b (127.1.221.129:40163)
I20260810 22:32:44.751741 21703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.129:46103
W20260810 22:32:44.868341 21906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:44.868631 21906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:44.868700 21906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:44.872756 21906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:44.872841 21906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:44.872927 21906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:32:44.877866 21906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:44.878983 21906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:44.880429 21906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:44.890275 21913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:44.890341 21906 server_base.cc:1034] running on GCE node
W20260810 22:32:44.890270 21915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:44.890342 21912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:44.890985 21906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:44.891619 21906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:44.892824 21906 hybrid_clock.cc:648] HybridClock initialized: now 1786401164892809 us; error 24 us; skew 500 ppm
I20260810 22:32:44.894737 21906 webserver.cc:460] Webserver started at http://127.1.221.130:36535/ using document root <none> and password file <none>
I20260810 22:32:44.895325 21906 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:44.895424 21906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:44.895658 21906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:44.897856 21906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "9e894f685324460ba66e2658d3f68d07"
format_stamp: "Formatted at 2026-08-10 22:32:44 on dist-test-slave-njxd"
server_key: "ed3cd8ce84ce95938d28b9ade9ed8ab9"
server_key_iv: "9689c2b7d41c1d1279af8129f9498da9"
server_key_version: "encryptionkey@0"
I20260810 22:32:44.898427 21906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "9e894f685324460ba66e2658d3f68d07"
format_stamp: "Formatted at 2026-08-10 22:32:44 on dist-test-slave-njxd"
server_key: "ed3cd8ce84ce95938d28b9ade9ed8ab9"
server_key_iv: "9689c2b7d41c1d1279af8129f9498da9"
server_key_version: "encryptionkey@0"
I20260810 22:32:44.902164 21906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:32:44.904500 21924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:44.905753 21906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:32:44.905881 21906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "9e894f685324460ba66e2658d3f68d07"
format_stamp: "Formatted at 2026-08-10 22:32:44 on dist-test-slave-njxd"
server_key: "ed3cd8ce84ce95938d28b9ade9ed8ab9"
server_key_iv: "9689c2b7d41c1d1279af8129f9498da9"
server_key_version: "encryptionkey@0"
I20260810 22:32:44.905992 21906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:44.930001 21906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:44.930686 21906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:44.930845 21906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:44.931414 21906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:44.932350 21906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:44.932399 21906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:44.932439 21906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:44.932454 21906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:44.941607 21906 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:40017
I20260810 22:32:44.941649 22052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:40017 every 8 connection(s)
I20260810 22:32:44.942534 21906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:32:44.946887 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 21906
I20260810 22:32:44.947018 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:32:44.947297 1910 external_mini_cluster.cc:1428] Setting key c716f2e4aee4bfb9a7029387c3c7a093
I20260810 22:32:44.949396 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43389
--builtin_ntp_servers=127.1.221.148:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 22:32:44.952343 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.944355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:36193 (local address 127.1.221.190:43389)
0810 22:32:44.944511 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:44.944514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:44.945565 (+ 1051us) server_negotiation.cc:407] Connection header received
0810 22:32:44.947020 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:44.947024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:44.947090 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:44.947200 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:44.948249 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.948937 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.950180 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.950571 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.950707 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.951823 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:44.951827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:44.951830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:44.951849 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:44.951868 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:44.951876 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:44.951955 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:44.952139 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 22:32:44.952204 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:32:44.953246 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:44.944768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43389 (local address 127.1.221.130:36193)
0810 22:32:44.945412 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:44.945452 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:44.946799 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:44.947386 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:44.947392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:44.947475 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:44.948102 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.948122 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.949076 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.949079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.950055 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:44.950065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:44.950739 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:44.950742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:44.950884 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:44.951558 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:32:44.951560 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:44.951607 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:32:44.951669 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:44.951959 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:44.951972 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:44.951978 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:32:44.952127 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 22:32:44.952499 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260810 22:32:44.954627 22054 heartbeater.cc:347] Connected to a master server at 127.1.221.190:43389
I20260810 22:32:44.954890 22054 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:44.955426 22054 heartbeater.cc:511] Master 127.1.221.190:43389 requested a full tablet report, sending...
I20260810 22:32:44.956597 21703 ts_manager.cc:194] Registered new tserver with Master: 9e894f685324460ba66e2658d3f68d07 (127.1.221.130:40017)
I20260810 22:32:44.957487 21703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.130:36193
W20260810 22:32:45.068717 22058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:45.068996 22058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:45.069036 22058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:45.075819 22058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:45.075953 22058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:45.076052 22058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:32:45.083374 22058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:43389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:45.084662 22058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:45.086202 22058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:45.096400 22063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:45.096352 22058 server_base.cc:1034] running on GCE node
W20260810 22:32:45.096347 22064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:45.096361 22066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:45.097105 22058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:45.097817 22058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:45.099013 22058 hybrid_clock.cc:648] HybridClock initialized: now 1786401165098995 us; error 31 us; skew 500 ppm
I20260810 22:32:45.100996 22058 webserver.cc:460] Webserver started at http://127.1.221.131:44207/ using document root <none> and password file <none>
I20260810 22:32:45.101780 22058 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:45.101884 22058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:45.102159 22058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:45.104413 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "d26c1df7838f4e76a25714dfc8c8b2a6"
format_stamp: "Formatted at 2026-08-10 22:32:45 on dist-test-slave-njxd"
server_key: "5cb05eb3fdc3fa99480e66848875da90"
server_key_iv: "feaa1c9b40b0887f500beca4d6a6640a"
server_key_version: "encryptionkey@0"
I20260810 22:32:45.105031 22058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "d26c1df7838f4e76a25714dfc8c8b2a6"
format_stamp: "Formatted at 2026-08-10 22:32:45 on dist-test-slave-njxd"
server_key: "5cb05eb3fdc3fa99480e66848875da90"
server_key_iv: "feaa1c9b40b0887f500beca4d6a6640a"
server_key_version: "encryptionkey@0"
I20260810 22:32:45.109275 22058 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 22:32:45.112043 22073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:45.113309 22058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:45.113471 22058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "d26c1df7838f4e76a25714dfc8c8b2a6"
format_stamp: "Formatted at 2026-08-10 22:32:45 on dist-test-slave-njxd"
server_key: "5cb05eb3fdc3fa99480e66848875da90"
server_key_iv: "feaa1c9b40b0887f500beca4d6a6640a"
server_key_version: "encryptionkey@0"
I20260810 22:32:45.113623 22058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:45.139142 22058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:45.139984 22058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:45.140235 22058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:45.140923 22058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:45.141975 22058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:45.142064 22058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:45.142140 22058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:45.142189 22058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:45.151790 22058 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:34781
I20260810 22:32:45.151825 22200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:34781 every 8 connection(s)
I20260810 22:32:45.152817 22058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:32:45.162652 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.154776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:35067 (local address 127.1.221.190:43389)
0810 22:32:45.154934 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:45.154938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:45.156102 (+ 1164us) server_negotiation.cc:407] Connection header received
0810 22:32:45.157641 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:45.157644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:45.157706 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:45.157824 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:45.158778 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.159327 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.160528 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.160761 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.160883 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.162098 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:45.162102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:45.162107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:45.162126 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:45.162145 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:45.162154 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:45.162232 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:45.162431 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 22:32:45.162499 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 22:32:45.162664 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 22058
I20260810 22:32:45.162815 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:32:45.163091 1910 external_mini_cluster.cc:1428] Setting key 769a7499d7e9d0b362244caea25ff0ba
I20260810 22:32:45.163695 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.155234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:43389 (local address 127.1.221.131:35067)
0810 22:32:45.155935 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:45.155977 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:45.157375 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:45.158013 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:45.158020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:45.158107 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:45.158619 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:45.158632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.159470 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.159478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:45.160389 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:45.160399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.160930 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.160934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:45.161109 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.161823 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:32:45.161825 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:45.161867 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:32:45.161933 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:45.162234 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:45.162248 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:45.162256 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:32:45.162410 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 22:32:45.162700 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":115,"threads_started":1}
I20260810 22:32:45.165055 22201 heartbeater.cc:347] Connected to a master server at 127.1.221.190:43389
I20260810 22:32:45.165354 22201 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:45.165927 22201 heartbeater.cc:511] Master 127.1.221.190:43389 requested a full tablet report, sending...
I20260810 22:32:45.166926 21702 ts_manager.cc:194] Registered new tserver with Master: d26c1df7838f4e76a25714dfc8c8b2a6 (127.1.221.131:34781)
I20260810 22:32:45.167471 21702 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.131:35067
I20260810 22:32:45.177408 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:32:45.183557 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.179533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.1.221.190:43389)
0810 22:32:45.179730 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:45.179734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:45.179897 (+ 163us) server_negotiation.cc:407] Connection header received
0810 22:32:45.180048 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:45.180051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:45.180104 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:45.180186 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:45.180456 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.180980 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.181945 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.182174 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.182280 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.183053 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:45.183057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:45.183060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:45.183090 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:45.183118 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:45.183126 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:45.183206 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:45.183340 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 22:32:45.183399 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 22:32:45.186527 21702 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:32:45.189462 21702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:32:45.204159 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.197896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.1.221.130:40017)
0810 22:32:45.198872 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:45.198887 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:45.198916 (+ 29us) server_negotiation.cc:407] Connection header received
0810 22:32:45.199207 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:45.199215 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:45.199412 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:45.199535 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:45.199919 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.201162 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.202701 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.203438 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.203558 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.203768 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:45.203885 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 22:32:45.203992 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":123,"threads_started":1}
I20260810 22:32:45.204145 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.197739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:40017 (local address 127.0.0.1:53370)
0810 22:32:45.198341 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:45.198391 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:45.199051 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:45.199565 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:45.199572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:45.199635 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:45.199780 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:45.199787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.201426 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.201432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:45.202578 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:45.202588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.203572 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.203576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:45.203693 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.203730 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 22:32:45.203852 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 22:32:45.203968 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":88,"thread_start_us":79,"threads_started":1}
I20260810 22:32:45.204452 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.198138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:40163 (local address 127.0.0.1:44360)
0810 22:32:45.198395 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:45.198412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:45.198825 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:45.200155 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:45.200158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:45.200174 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:45.200272 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:45.200279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.201275 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.201279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:45.202846 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:45.202857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.203862 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.203865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:45.203965 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.203991 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:32:45.204041 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 22:32:45.204113 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":133,"thread_start_us":103,"threads_started":1}
I20260810 22:32:45.204845 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.198719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.1.221.131:34781)
0810 22:32:45.199314 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:45.199322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:45.199341 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:32:45.199402 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:45.199405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:45.199604 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:45.199774 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:45.200005 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.201155 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.203023 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.203561 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.203676 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.203861 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:45.204225 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 22:32:45.204568 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":151,"threads_started":1}
I20260810 22:32:45.205189 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.198524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:34781 (local address 127.0.0.1:46456)
0810 22:32:45.199052 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:45.199065 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:45.199145 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:45.199768 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:45.199771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:45.199800 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:45.199886 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:45.199892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.201289 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.201294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:45.202929 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:45.202939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.203919 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.203923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:45.204000 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.204020 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 22:32:45.204085 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 22:32:45.204146 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":177,"thread_start_us":77,"threads_started":1}
I20260810 22:32:45.205767 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.198702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.1.221.129:40163)
0810 22:32:45.199733 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:45.199747 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:45.199770 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:32:45.199845 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:45.199850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:45.200034 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:45.200177 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:45.200394 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.201115 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.202969 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.203716 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.204223 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.204424 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:45.204521 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 22:32:45.205610 (+ 1089us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":124,"threads_started":1}
I20260810 22:32:45.207931 21832 tablet_service.cc:1467] Processing CreateTablet for tablet ae1d6873297a4ddaa030676e14252b23 (DEFAULT_TABLE table=test-table [id=db6d4ea4fa134867952434e81857abe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:45.207842 21982 tablet_service.cc:1467] Processing CreateTablet for tablet ae1d6873297a4ddaa030676e14252b23 (DEFAULT_TABLE table=test-table [id=db6d4ea4fa134867952434e81857abe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:45.207842 22124 tablet_service.cc:1467] Processing CreateTablet for tablet ae1d6873297a4ddaa030676e14252b23 (DEFAULT_TABLE table=test-table [id=db6d4ea4fa134867952434e81857abe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:45.208953 22124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1d6873297a4ddaa030676e14252b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:45.208981 21832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1d6873297a4ddaa030676e14252b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:45.208953 21982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae1d6873297a4ddaa030676e14252b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:45.215338 22218 tablet_bootstrap.cc:492] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b: Bootstrap starting.
I20260810 22:32:45.216712 22219 tablet_bootstrap.cc:492] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6: Bootstrap starting.
I20260810 22:32:45.217912 22218 tablet_bootstrap.cc:654] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:45.218537 22219 tablet_bootstrap.cc:654] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:45.218732 22218 log.cc:824] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:45.219360 22219 log.cc:824] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:45.219736 22220 tablet_bootstrap.cc:492] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07: Bootstrap starting.
I20260810 22:32:45.221735 22219 tablet_bootstrap.cc:492] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6: No bootstrap required, opened a new log
I20260810 22:32:45.221974 22219 ts_tablet_manager.cc:1397] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 22:32:45.228111 22219 raft_consensus.cc:348] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e894f685324460ba66e2658d3f68d07" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40017 } } peers { permanent_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34781 } } peers { permanent_uuid: "66aefc22b0dc490d87b3e29928d87c4b" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40163 } }
I20260810 22:32:45.228567 22219 raft_consensus.cc:374] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:45.228689 22219 raft_consensus.cc:729] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26c1df7838f4e76a25714dfc8c8b2a6, State: Initialized, Role: FOLLOWER
I20260810 22:32:45.229390 22219 consensus_queue.cc:260] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e894f685324460ba66e2658d3f68d07" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40017 } } peers { permanent_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34781 } } peers { permanent_uuid: "66aefc22b0dc490d87b3e29928d87c4b" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40163 } }
I20260810 22:32:45.230135 22218 tablet_bootstrap.cc:492] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b: No bootstrap required, opened a new log
I20260810 22:32:45.230369 22218 ts_tablet_manager.cc:1397] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260810 22:32:45.230803 22219 ts_tablet_manager.cc:1428] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260810 22:32:45.230981 22220 tablet_bootstrap.cc:654] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:45.231299 22201 heartbeater.cc:503] Master 127.1.221.190:43389 was elected leader, sending a full tablet report...
I20260810 22:32:45.232635 22220 log.cc:824] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:45.235118 22220 tablet_bootstrap.cc:492] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07: No bootstrap required, opened a new log
I20260810 22:32:45.235334 22220 ts_tablet_manager.cc:1397] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260810 22:32:45.236042 22218 raft_consensus.cc:348] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e894f685324460ba66e2658d3f68d07" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40017 } } peers { permanent_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34781 } } peers { permanent_uuid: "66aefc22b0dc490d87b3e29928d87c4b" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40163 } }
I20260810 22:32:45.236321 22218 raft_consensus.cc:374] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:45.236385 22218 raft_consensus.cc:729] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66aefc22b0dc490d87b3e29928d87c4b, State: Initialized, Role: FOLLOWER
I20260810 22:32:45.236874 22218 consensus_queue.cc:260] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e894f685324460ba66e2658d3f68d07" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40017 } } peers { permanent_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34781 } } peers { permanent_uuid: "66aefc22b0dc490d87b3e29928d87c4b" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40163 } }
I20260810 22:32:45.238029 21903 heartbeater.cc:503] Master 127.1.221.190:43389 was elected leader, sending a full tablet report...
I20260810 22:32:45.237862 22218 ts_tablet_manager.cc:1428] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 22:32:45.241336 22220 raft_consensus.cc:348] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e894f685324460ba66e2658d3f68d07" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40017 } } peers { permanent_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34781 } } peers { permanent_uuid: "66aefc22b0dc490d87b3e29928d87c4b" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40163 } }
I20260810 22:32:45.241672 22220 raft_consensus.cc:374] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:45.241983 22220 raft_consensus.cc:729] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e894f685324460ba66e2658d3f68d07, State: Initialized, Role: FOLLOWER
I20260810 22:32:45.242509 22220 consensus_queue.cc:260] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e894f685324460ba66e2658d3f68d07" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40017 } } peers { permanent_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34781 } } peers { permanent_uuid: "66aefc22b0dc490d87b3e29928d87c4b" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40163 } }
I20260810 22:32:45.243232 22054 heartbeater.cc:503] Master 127.1.221.190:43389 was elected leader, sending a full tablet report...
I20260810 22:32:45.244102 22220 ts_tablet_manager.cc:1428] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
W20260810 22:32:45.403916 22202 tablet.cc:2369] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:32:45.431267 22223 raft_consensus.cc:482] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:32:45.431519 22223 raft_consensus.cc:504] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e894f685324460ba66e2658d3f68d07" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40017 } } peers { permanent_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34781 } } peers { permanent_uuid: "66aefc22b0dc490d87b3e29928d87c4b" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40163 } }
I20260810 22:32:45.432801 22223 leader_election.cc:288] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e894f685324460ba66e2658d3f68d07 (127.1.221.130:40017), 66aefc22b0dc490d87b3e29928d87c4b (127.1.221.129:40163)
I20260810 22:32:45.437616 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.433132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:40163 (local address 127.1.221.131:52885)
0810 22:32:45.433345 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:45.433362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:45.433474 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:45.433919 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:45.433923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:45.433946 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:45.434089 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:45.434099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.435043 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.435048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:45.435995 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:45.436006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.437138 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.437142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:45.437250 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.437292 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 22:32:45.437364 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 22:32:45.437453 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260810 22:32:45.438062 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.433572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:43595 (local address 127.1.221.130:40017)
0810 22:32:45.433757 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:45.433761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:45.433904 (+ 143us) server_negotiation.cc:407] Connection header received
0810 22:32:45.434143 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:45.434148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:45.434223 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:45.434318 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:45.434637 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.435563 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.436742 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.437221 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.437359 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.437424 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:45.437762 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 22:32:45.437881 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260810 22:32:45.438139 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.433213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:52885 (local address 127.1.221.129:40163)
0810 22:32:45.433346 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:45.433352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:45.433556 (+ 204us) server_negotiation.cc:407] Connection header received
0810 22:32:45.433713 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:45.433718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:45.433795 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:45.433887 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:45.434241 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.434923 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.436155 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.436981 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.437140 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.437222 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:45.437908 (+ 686us) server_negotiation.cc:299] Negotiation successful
0810 22:32:45.438000 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:32:45.438560 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.433220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:40017 (local address 127.1.221.131:43595)
0810 22:32:45.433797 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:45.433815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:45.433962 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:45.434340 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:45.434342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:45.434361 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:45.434485 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:45.434492 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.435728 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.435732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:45.436607 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:45.436617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.437376 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.437383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:45.437537 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.437611 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 22:32:45.438238 (+ 627us) client_negotiation.cc:240] Negotiation successful
0810 22:32:45.438350 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":132,"threads_started":1}
I20260810 22:32:45.438791 21855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1d6873297a4ddaa030676e14252b23" candidate_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66aefc22b0dc490d87b3e29928d87c4b" is_pre_election: true
I20260810 22:32:45.439154 21855 raft_consensus.cc:2454] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26c1df7838f4e76a25714dfc8c8b2a6 in term 0.
I20260810 22:32:45.439384 22004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1d6873297a4ddaa030676e14252b23" candidate_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e894f685324460ba66e2658d3f68d07" is_pre_election: true
I20260810 22:32:45.439740 22074 leader_election.cc:302] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66aefc22b0dc490d87b3e29928d87c4b, d26c1df7838f4e76a25714dfc8c8b2a6; no voters:
I20260810 22:32:45.439792 22004 raft_consensus.cc:2454] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d26c1df7838f4e76a25714dfc8c8b2a6 in term 0.
I20260810 22:32:45.440099 22223 raft_consensus.cc:2781] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:32:45.440266 22223 raft_consensus.cc:482] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:32:45.440338 22223 raft_consensus.cc:3037] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:45.441396 22223 raft_consensus.cc:504] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e894f685324460ba66e2658d3f68d07" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40017 } } peers { permanent_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34781 } } peers { permanent_uuid: "66aefc22b0dc490d87b3e29928d87c4b" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40163 } }
I20260810 22:32:45.441846 22223 leader_election.cc:288] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [CANDIDATE]: Term 1 election: Requested vote from peers 9e894f685324460ba66e2658d3f68d07 (127.1.221.130:40017), 66aefc22b0dc490d87b3e29928d87c4b (127.1.221.129:40163)
I20260810 22:32:45.442224 22004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1d6873297a4ddaa030676e14252b23" candidate_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e894f685324460ba66e2658d3f68d07"
I20260810 22:32:45.442281 21855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae1d6873297a4ddaa030676e14252b23" candidate_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66aefc22b0dc490d87b3e29928d87c4b"
I20260810 22:32:45.442394 22004 raft_consensus.cc:3037] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:45.442423 21855 raft_consensus.cc:3037] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:45.443635 21855 raft_consensus.cc:2454] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26c1df7838f4e76a25714dfc8c8b2a6 in term 1.
W20260810 22:32:45.443656 22055 tablet.cc:2369] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:32:45.444001 22074 leader_election.cc:302] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66aefc22b0dc490d87b3e29928d87c4b, d26c1df7838f4e76a25714dfc8c8b2a6; no voters:
I20260810 22:32:45.444243 22223 raft_consensus.cc:2781] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:45.444465 22004 raft_consensus.cc:2454] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d26c1df7838f4e76a25714dfc8c8b2a6 in term 1.
I20260810 22:32:45.444514 22223 raft_consensus.cc:686] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [term 1 LEADER]: Becoming Leader. State: Replica: d26c1df7838f4e76a25714dfc8c8b2a6, State: Running, Role: LEADER
I20260810 22:32:45.445015 22223 consensus_queue.cc:237] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e894f685324460ba66e2658d3f68d07" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40017 } } peers { permanent_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34781 } } peers { permanent_uuid: "66aefc22b0dc490d87b3e29928d87c4b" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40163 } }
I20260810 22:32:45.448665 21703 catalog_manager.cc:5495] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d26c1df7838f4e76a25714dfc8c8b2a6 (127.1.221.131). New cstate: current_term: 1 leader_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e894f685324460ba66e2658d3f68d07" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d26c1df7838f4e76a25714dfc8c8b2a6" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66aefc22b0dc490d87b3e29928d87c4b" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40163 } health_report { overall_health: UNKNOWN } } }
W20260810 22:32:45.487469 21904 tablet.cc:2369] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:32:45.505025 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.500990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.1.221.131:34781)
0810 22:32:45.501152 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:45.501157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:45.501175 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:32:45.501299 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:45.501302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:45.501374 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:45.501479 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:32:45.501792 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.502573 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.503416 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:45.503774 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:45.503878 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:45.504151 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:32:45.504652 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:32:45.504783 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:45.504816 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 22:32:45.504867 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:32:45.511659 22004 raft_consensus.cc:1261] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07 [term 1 FOLLOWER]: Refusing update from remote peer d26c1df7838f4e76a25714dfc8c8b2a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:45.511869 21855 raft_consensus.cc:1261] T ae1d6873297a4ddaa030676e14252b23 P 66aefc22b0dc490d87b3e29928d87c4b [term 1 FOLLOWER]: Refusing update from remote peer d26c1df7838f4e76a25714dfc8c8b2a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:45.512521 22223 consensus_queue.cc:1035] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66aefc22b0dc490d87b3e29928d87c4b" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:32:45.512740 22229 consensus_queue.cc:1035] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e894f685324460ba66e2658d3f68d07" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 40017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:32:45.520183 22233 mvcc.cc:205] Tried to move back new op lower bound from 7317099173926924288 to 7317099173665751040. Current Snapshot: MvccSnapshot[applied={T|T < 7317099173926924288}]
I20260810 22:32:45.522413 22234 mvcc.cc:205] Tried to move back new op lower bound from 7317099173926924288 to 7317099173665751040. Current Snapshot: MvccSnapshot[applied={T|T < 7317099173926924288}]
I20260810 22:32:45.528798 21702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:45.528990 21702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:45.531540 21702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2a96164a5e64a07b860562ac1783a1f: Sending DeleteTablet for 3 replicas of tablet ae1d6873297a4ddaa030676e14252b23
I20260810 22:32:45.532351 21832 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1d6873297a4ddaa030676e14252b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:45 UTC) from {username='slave'} at 127.0.0.1:44360
I20260810 22:32:45.532578 21982 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1d6873297a4ddaa030676e14252b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:45 UTC) from {username='slave'} at 127.0.0.1:53370
I20260810 22:32:45.532902 22124 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1d6873297a4ddaa030676e14252b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:45 UTC) from {username='slave'} at 127.0.0.1:46456
I20260810 22:32:45.533038 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 21766
I20260810 22:32:45.536480 22245 tablet_replica.cc:330] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07: stopping tablet replica
I20260810 22:32:45.536890 22245 raft_consensus.cc:2229] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:32:45.537346 22245 raft_consensus.cc:2258] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:45.537834 22246 tablet_replica.cc:330] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6: stopping tablet replica
I20260810 22:32:45.538162 22246 raft_consensus.cc:2229] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:32:45.538614 22246 raft_consensus.cc:2258] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:45.539450 22245 ts_tablet_manager.cc:1905] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:45.540134 22246 ts_tablet_manager.cc:1905] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:45.542570 22245 ts_tablet_manager.cc:1918] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:45.542761 22245 log.cc:1196] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/ae1d6873297a4ddaa030676e14252b23
I20260810 22:32:45.543159 22245 ts_tablet_manager.cc:1939] T ae1d6873297a4ddaa030676e14252b23 P 9e894f685324460ba66e2658d3f68d07: Deleting consensus metadata
I20260810 22:32:45.543766 22246 ts_tablet_manager.cc:1918] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:45.543879 22246 log.cc:1196] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/ae1d6873297a4ddaa030676e14252b23
I20260810 22:32:45.544239 22246 ts_tablet_manager.cc:1939] T ae1d6873297a4ddaa030676e14252b23 P d26c1df7838f4e76a25714dfc8c8b2a6: Deleting consensus metadata
I20260810 22:32:45.544843 21688 catalog_manager.cc:4841] TS 9e894f685324460ba66e2658d3f68d07 (127.1.221.130:40017): tablet ae1d6873297a4ddaa030676e14252b23 (table test-table [id=db6d4ea4fa134867952434e81857abe3]) successfully deleted
W20260810 22:32:45.545267 21685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:40163 (error 108)
I20260810 22:32:45.545354 21688 catalog_manager.cc:4841] TS d26c1df7838f4e76a25714dfc8c8b2a6 (127.1.221.131:34781): tablet ae1d6873297a4ddaa030676e14252b23 (table test-table [id=db6d4ea4fa134867952434e81857abe3]) successfully deleted
I20260810 22:32:45.545791 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 21906
I20260810 22:32:45.546905 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:45.546560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:40163 (local address 127.0.0.1:44376)
0810 22:32:45.546745 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:45.546814 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:40163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260810 22:32:45.547237 21685 catalog_manager.cc:4584] TS 66aefc22b0dc490d87b3e29928d87c4b (127.1.221.129:40163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae1d6873297a4ddaa030676e14252b23: Network error: Client connection negotiation failed: client connection to 127.1.221.129:40163: connect: Connection refused (error 111)
I20260810 22:32:45.555570 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 22058
I20260810 22:32:45.563730 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 21666
2026-08-10T22:32:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1273 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:32:45 dist-test-slave-njxd krb5kdc[22250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:32:45 dist-test-slave-njxd krb5kdc[22250](info): set up 2 sockets
Aug 10 22:32:45 dist-test-slave-njxd krb5kdc[22250](info): commencing operation
krb5kdc: starting...
W20260810 22:32:46.666949 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:32:46 dist-test-slave-njxd krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:32:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:32:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:32:46.800719 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:45111
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:46.914166 22264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:46.914458 22264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:46.914538 22264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:46.918594 22264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:32:46.918680 22264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:46.918720 22264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:46.918777 22264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:32:46.918815 22264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:32:46.924124 22264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:45111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:46.925276 22264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:46.926393 22264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:46.935034 22273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:46.935042 22272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:46.935042 22275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:46.935325 22264 server_base.cc:1034] running on GCE node
I20260810 22:32:46.936004 22264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:46.937227 22264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:46.938421 22264 hybrid_clock.cc:648] HybridClock initialized: now 1786401166938381 us; error 62 us; skew 500 ppm
Aug 10 22:32:46 dist-test-slave-njxd krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401166, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:46.947557 22264 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:32:46.948596 22264 webserver.cc:460] Webserver started at http://127.1.221.190:42661/ using document root <none> and password file <none>
I20260810 22:32:46.949173 22264 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:46.949239 22264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:46.949417 22264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:46.951687 22264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "1c998d6b1a3d457b885e97deb1326ced"
format_stamp: "Formatted at 2026-08-10 22:32:46 on dist-test-slave-njxd"
server_key: "ce77972dfe09d119186cc33103b3b444"
server_key_iv: "baf2f7cf5e5459cfb5466241850b44e8"
server_key_version: "encryptionkey@0"
I20260810 22:32:46.952244 22264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "1c998d6b1a3d457b885e97deb1326ced"
format_stamp: "Formatted at 2026-08-10 22:32:46 on dist-test-slave-njxd"
server_key: "ce77972dfe09d119186cc33103b3b444"
server_key_iv: "baf2f7cf5e5459cfb5466241850b44e8"
server_key_version: "encryptionkey@0"
I20260810 22:32:46.956233 22264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:32:46.958905 22282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:46.960462 22264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:32:46.960577 22264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "1c998d6b1a3d457b885e97deb1326ced"
format_stamp: "Formatted at 2026-08-10 22:32:46 on dist-test-slave-njxd"
server_key: "ce77972dfe09d119186cc33103b3b444"
server_key_iv: "baf2f7cf5e5459cfb5466241850b44e8"
server_key_version: "encryptionkey@0"
I20260810 22:32:46.960747 22264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:46.972749 22264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:46.976255 22264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:46.976483 22264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:46.984208 22264 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:45111
I20260810 22:32:46.984244 22342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:45111 every 8 connection(s)
I20260810 22:32:46.985209 22264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:32:46.988111 22343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:46.988149 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 22264
I20260810 22:32:46.988304 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:32:46.988653 1910 external_mini_cluster.cc:1428] Setting key e45dbd07d423fb333246e91b29999e6e
I20260810 22:32:46.994752 22343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced: Bootstrap starting.
I20260810 22:32:46.997018 22343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:46.997823 22343 log.cc:824] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:32:46 dist-test-slave-njxd krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:32:47.000257 22343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced: No bootstrap required, opened a new log
I20260810 22:32:47.003840 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:46.990182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53246 (local address 127.1.221.190:45111)
0810 22:32:46.990653 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:46.990669 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:46.990707 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:32:46.991554 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:46.991575 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:46.991905 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:46.992252 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:46.993092 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:46.994175 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:46.995120 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:46.995593 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:46.995774 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:46.998499 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:46.998504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:46.998517 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:46.998542 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:47.000533 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:47.000993 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:47.001000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:47.001006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:47.001153 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:47.001413 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:47.001417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:47.001419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:47.001976 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:47.002190 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:47.002415 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 22:32:47.002803 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":147,"threads_started":1}
I20260810 22:32:47.004998 22343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c998d6b1a3d457b885e97deb1326ced" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45111 } }
I20260810 22:32:47.005280 22343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:47.005338 22343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c998d6b1a3d457b885e97deb1326ced, State: Initialized, Role: FOLLOWER
I20260810 22:32:47.005821 22343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c998d6b1a3d457b885e97deb1326ced" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45111 } }
I20260810 22:32:47.005976 22343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:32:47.006048 22343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:32:47.006165 22343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:47.007135 22343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c998d6b1a3d457b885e97deb1326ced" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45111 } }
I20260810 22:32:47.007493 22343 leader_election.cc:302] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c998d6b1a3d457b885e97deb1326ced; no voters:
I20260810 22:32:47.007876 22343 leader_election.cc:288] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:32:47.007982 22348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:47.008196 22348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [term 1 LEADER]: Becoming Leader. State: Replica: 1c998d6b1a3d457b885e97deb1326ced, State: Running, Role: LEADER
I20260810 22:32:47.008546 22348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c998d6b1a3d457b885e97deb1326ced" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45111 } }
I20260810 22:32:47.009243 22343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:32:47.010324 22350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c998d6b1a3d457b885e97deb1326ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c998d6b1a3d457b885e97deb1326ced" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45111 } } }
I20260810 22:32:47.010457 22350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:47.010866 22357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:32:47.010846 22349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c998d6b1a3d457b885e97deb1326ced. Latest consensus state: current_term: 1 leader_uuid: "1c998d6b1a3d457b885e97deb1326ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c998d6b1a3d457b885e97deb1326ced" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45111 } } }
I20260810 22:32:47.010993 22349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:47.013901 22357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:32:47.019502 22357 catalog_manager.cc:1347] Generated new cluster ID: 35d64f9bbbe34c3d9ed56701163324d3
I20260810 22:32:47.019590 22357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:32:47.033484 22357 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:32:47.035072 22357 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:32:47.047930 22357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced: Generated new TSK 0
I20260810 22:32:47.048820 22357 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:32:47.082795 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45111
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:47.198019 22369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:47.198282 22369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:47.198324 22369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:47.202327 22369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:47.202385 22369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:47.202476 22369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:32:47.208379 22369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:47.209518 22369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:47.210836 22369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:47.220971 22378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:47.220956 22374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:47.221117 22369 server_base.cc:1034] running on GCE node
W20260810 22:32:47.221329 22375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:47.221871 22369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:47.222776 22369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:47.223968 22369 hybrid_clock.cc:648] HybridClock initialized: now 1786401167223949 us; error 34 us; skew 500 ppm
Aug 10 22:32:47 dist-test-slave-njxd krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:47.226897 22369 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:32:47.227836 22369 webserver.cc:460] Webserver started at http://127.1.221.129:37873/ using document root <none> and password file <none>
I20260810 22:32:47.228591 22369 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:47.228690 22369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:47.228902 22369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:47.231051 22369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "b4244d36d0f1417c970e077340a6b469"
format_stamp: "Formatted at 2026-08-10 22:32:47 on dist-test-slave-njxd"
server_key: "3c8cafe84d85825a2b6a850e8efadd7e"
server_key_iv: "286a77d775f4983d91ce5ae6e4b324dd"
server_key_version: "encryptionkey@0"
I20260810 22:32:47.231607 22369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "b4244d36d0f1417c970e077340a6b469"
format_stamp: "Formatted at 2026-08-10 22:32:47 on dist-test-slave-njxd"
server_key: "3c8cafe84d85825a2b6a850e8efadd7e"
server_key_iv: "286a77d775f4983d91ce5ae6e4b324dd"
server_key_version: "encryptionkey@0"
I20260810 22:32:47.235466 22369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:32:47.237957 22388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:47.239235 22369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:47.239362 22369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "b4244d36d0f1417c970e077340a6b469"
format_stamp: "Formatted at 2026-08-10 22:32:47 on dist-test-slave-njxd"
server_key: "3c8cafe84d85825a2b6a850e8efadd7e"
server_key_iv: "286a77d775f4983d91ce5ae6e4b324dd"
server_key_version: "encryptionkey@0"
I20260810 22:32:47.239478 22369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:47.277891 22369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:47.280859 22369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:47.281075 22369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:47.281793 22369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:47.282687 22369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:47.282734 22369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:47.282807 22369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:47.282838 22369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:47.292477 22369 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:37053
I20260810 22:32:47.292510 22514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:37053 every 8 connection(s)
I20260810 22:32:47.293388 22369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:32:47.300856 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 22369
I20260810 22:32:47.301008 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:32:47.301317 1910 external_mini_cluster.cc:1428] Setting key 16a685c267afa8700140af24a4d0f754
Aug 10 22:32:47 dist-test-slave-njxd krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:32:47.307276 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.295207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:38767 (local address 127.1.221.190:45111)
0810 22:32:47.295352 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:47.295355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:47.296247 (+ 892us) server_negotiation.cc:407] Connection header received
0810 22:32:47.297162 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:47.297165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:47.297217 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:47.297293 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:47.298608 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.299121 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.300239 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.300471 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.300546 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.303468 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:47.303470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:47.303472 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:47.303496 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:47.305223 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:47.305756 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:47.305759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:47.305761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:47.305858 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:47.306192 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:47.306198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:47.306203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:47.306395 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:47.306491 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:47.306978 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 22:32:47.307103 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:32:47.308105 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.295566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:45111 (local address 127.1.221.129:38767)
0810 22:32:47.296029 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:47.296081 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:47.296949 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:47.297503 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:47.297510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:47.297971 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:47.298465 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.298479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.299238 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.299241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.300126 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.300135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.300645 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.300648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.300805 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.301449 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:47.301469 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:47.303286 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:47.305341 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:47.305346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:47.305358 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:47.305661 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:47.305955 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:47.305957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:47.305959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:47.306089 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:47.306508 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:47.306513 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:47.306737 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 22:32:47.306918 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 22:32:47.307247 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260810 22:32:47.309404 22515 heartbeater.cc:347] Connected to a master server at 127.1.221.190:45111
I20260810 22:32:47.309680 22515 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:47.310281 22515 heartbeater.cc:511] Master 127.1.221.190:45111 requested a full tablet report, sending...
I20260810 22:32:47.311838 22301 ts_manager.cc:194] Registered new tserver with Master: b4244d36d0f1417c970e077340a6b469 (127.1.221.129:37053)
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260810 22:32:47.312981 22301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:38767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:32:47.332747 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45111
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:47.446276 22520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:47.446606 22520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:47.446689 22520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:47.450909 22520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:47.451006 22520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:47.451109 22520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:32:47.456593 22520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:47.457866 22520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:47.459048 22520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:47.469193 22525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:47.469244 22529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:47.469352 22520 server_base.cc:1034] running on GCE node
W20260810 22:32:47.469547 22527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:47.470034 22520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:47.470700 22520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:47.471897 22520 hybrid_clock.cc:648] HybridClock initialized: now 1786401167471877 us; error 35 us; skew 500 ppm
Aug 10 22:32:47 dist-test-slave-njxd krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:47.474892 22520 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:32:47.475759 22520 webserver.cc:460] Webserver started at http://127.1.221.130:36345/ using document root <none> and password file <none>
I20260810 22:32:47.476292 22520 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:47.476351 22520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:47.476523 22520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:47.478689 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "63b86e6c722348428ce678b2c8cd86ac"
format_stamp: "Formatted at 2026-08-10 22:32:47 on dist-test-slave-njxd"
server_key: "95cfadcbae3e3dfb6c45f9d22f765acb"
server_key_iv: "9760dda7a00e112f24ed218286570728"
server_key_version: "encryptionkey@0"
I20260810 22:32:47.479307 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "63b86e6c722348428ce678b2c8cd86ac"
format_stamp: "Formatted at 2026-08-10 22:32:47 on dist-test-slave-njxd"
server_key: "95cfadcbae3e3dfb6c45f9d22f765acb"
server_key_iv: "9760dda7a00e112f24ed218286570728"
server_key_version: "encryptionkey@0"
I20260810 22:32:47.483191 22520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:32:47.485780 22536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:47.487062 22520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:32:47.487195 22520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "63b86e6c722348428ce678b2c8cd86ac"
format_stamp: "Formatted at 2026-08-10 22:32:47 on dist-test-slave-njxd"
server_key: "95cfadcbae3e3dfb6c45f9d22f765acb"
server_key_iv: "9760dda7a00e112f24ed218286570728"
server_key_version: "encryptionkey@0"
I20260810 22:32:47.487317 22520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:47.503361 22520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:47.506410 22520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:47.506630 22520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:47.507263 22520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:47.508119 22520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:47.508167 22520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:47.508242 22520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:47.508279 22520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:47.517814 22520 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:37823
I20260810 22:32:47.517834 22659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:37823 every 8 connection(s)
I20260810 22:32:47.518796 22520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:32:47.519698 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 22520
I20260810 22:32:47.519802 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:32:47.520109 1910 external_mini_cluster.cc:1428] Setting key bfe587e1841417d1466fd3f8055c70e1
Aug 10 22:32:47 dist-test-slave-njxd krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260810 22:32:47.534888 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.520872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:56049 (local address 127.1.221.190:45111)
0810 22:32:47.521016 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:47.521020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:47.522144 (+ 1124us) server_negotiation.cc:407] Connection header received
0810 22:32:47.523542 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:47.523544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:47.523591 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:47.523681 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:47.525062 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.525757 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.527675 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.527897 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.527978 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.531072 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:47.531075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:47.531077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:47.531103 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:47.532644 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:47.533204 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:47.533207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:47.533209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:47.533297 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:47.533696 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:47.533700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:47.533702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:47.533901 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:47.534012 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:47.534575 (+ 563us) server_negotiation.cc:299] Negotiation successful
0810 22:32:47.534716 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 22:32:47.535888 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.521343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:45111 (local address 127.1.221.130:56049)
0810 22:32:47.521957 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:47.522022 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:47.523294 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:47.523925 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:47.523938 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:47.524382 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:47.524893 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.524908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.525897 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.525904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.527532 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.527549 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.528079 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.528085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.528349 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.529044 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:47.529073 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:47.530875 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:47.532777 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:47.532784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:47.532803 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:47.533082 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:47.533394 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:47.533397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:47.533399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:47.533558 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:47.534014 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:47.534023 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:47.534306 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 22:32:47.534505 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 22:32:47.534897 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":160,"threads_started":1}
I20260810 22:32:47.537305 22660 heartbeater.cc:347] Connected to a master server at 127.1.221.190:45111
I20260810 22:32:47.537636 22660 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:47.538304 22660 heartbeater.cc:511] Master 127.1.221.190:45111 requested a full tablet report, sending...
I20260810 22:32:47.539525 22301 ts_manager.cc:194] Registered new tserver with Master: 63b86e6c722348428ce678b2c8cd86ac (127.1.221.130:37823)
I20260810 22:32:47.540449 22301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:56049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:32:47.553153 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45111
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:43579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:47.670202 22665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:47.670480 22665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:47.670523 22665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:47.674613 22665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:47.674683 22665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:47.674747 22665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:32:47.680159 22665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:47.681298 22665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:47.682411 22665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:47.692298 22671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:47.692305 22672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:47.692305 22674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:47.692513 22665 server_base.cc:1034] running on GCE node
I20260810 22:32:47.693029 22665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:47.693730 22665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:47.694917 22665 hybrid_clock.cc:648] HybridClock initialized: now 1786401167694901 us; error 29 us; skew 500 ppm
Aug 10 22:32:47 dist-test-slave-njxd krb5kdc[22250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:47.697691 22665 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:32:47.698478 22665 webserver.cc:460] Webserver started at http://127.1.221.131:42955/ using document root <none> and password file <none>
I20260810 22:32:47.698988 22665 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:47.699047 22665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:47.699218 22665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:47.701134 22665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "d4ee890acc45466caea72447b3510fde"
format_stamp: "Formatted at 2026-08-10 22:32:47 on dist-test-slave-njxd"
server_key: "304ec111f895b0b5de4629895c600d4d"
server_key_iv: "acb3a92b403c18ea9c69b1cfea2e1e85"
server_key_version: "encryptionkey@0"
I20260810 22:32:47.701719 22665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "d4ee890acc45466caea72447b3510fde"
format_stamp: "Formatted at 2026-08-10 22:32:47 on dist-test-slave-njxd"
server_key: "304ec111f895b0b5de4629895c600d4d"
server_key_iv: "acb3a92b403c18ea9c69b1cfea2e1e85"
server_key_version: "encryptionkey@0"
I20260810 22:32:47.705353 22665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:32:47.707746 22682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:47.708992 22665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:32:47.709108 22665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "d4ee890acc45466caea72447b3510fde"
format_stamp: "Formatted at 2026-08-10 22:32:47 on dist-test-slave-njxd"
server_key: "304ec111f895b0b5de4629895c600d4d"
server_key_iv: "acb3a92b403c18ea9c69b1cfea2e1e85"
server_key_version: "encryptionkey@0"
I20260810 22:32:47.709223 22665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:47.727730 22665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:47.731348 22665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:47.731561 22665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:47.732197 22665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:47.733045 22665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:47.733093 22665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:47.733158 22665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:47.733189 22665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:47.742200 22665 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:43109
I20260810 22:32:47.742229 22808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:43109 every 8 connection(s)
I20260810 22:32:47.743218 22665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:32:47.751423 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 22665
I20260810 22:32:47.751577 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:32:47.751919 1910 external_mini_cluster.cc:1428] Setting key 1a64eb3bd2bf9a9ff46c03a3764a2767
Aug 10 22:32:47 dist-test-slave-njxd krb5kdc[22250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401167, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:32:47.756337 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.745023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:36319 (local address 127.1.221.190:45111)
0810 22:32:47.745205 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:47.745210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:47.746128 (+ 918us) server_negotiation.cc:407] Connection header received
0810 22:32:47.746982 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:47.746985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:47.747033 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:47.747109 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:47.748249 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.748774 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.749963 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.750182 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.750261 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.752839 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:47.752845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:47.752850 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:47.752879 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:47.754589 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:47.755047 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:47.755049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:47.755051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:47.755122 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:47.755408 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:47.755410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:47.755411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:47.755579 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:47.755654 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:47.756104 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 22:32:47.756221 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 22:32:47.757097 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.745444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:45111 (local address 127.1.221.131:36319)
0810 22:32:47.745971 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:47.746015 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:47.746786 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:47.747281 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:47.747288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:47.747638 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:47.748121 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.748133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.748885 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.748887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.749856 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.749865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.750311 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.750314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.750448 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.750996 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:47.751012 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:47.752648 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:47.754694 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:47.754698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:47.754715 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:47.754964 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:47.755202 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:47.755204 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:47.755206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:47.755331 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:47.755665 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:47.755670 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:47.755890 (+ 220us) client_negotiation.cc:769] Sending connection context
0810 22:32:47.756040 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 22:32:47.756276 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":181,"threads_started":1}
I20260810 22:32:47.758196 22809 heartbeater.cc:347] Connected to a master server at 127.1.221.190:45111
I20260810 22:32:47.758462 22809 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:47.758960 22809 heartbeater.cc:511] Master 127.1.221.190:45111 requested a full tablet report, sending...
I20260810 22:32:47.759963 22301 ts_manager.cc:194] Registered new tserver with Master: d4ee890acc45466caea72447b3510fde (127.1.221.131:43109)
I20260810 22:32:47.760517 22301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:36319
I20260810 22:32:47.766010 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:32:47.775696 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.768033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.1.221.190:45111)
0810 22:32:47.768171 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:47.768178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:47.768374 (+ 196us) server_negotiation.cc:407] Connection header received
0810 22:32:47.768474 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:47.768477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:47.768516 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:47.768583 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:47.769273 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.769876 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.770681 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.770953 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.771034 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.772558 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:47.772561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:47.772563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:47.772587 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:47.774121 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:47.774589 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:47.774592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:47.774598 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:47.774674 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:47.774944 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:47.774946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:47.774947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:47.775160 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:47.775277 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:47.775494 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 22:32:47.775573 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:32:47.778887 22301 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:32:47.782029 22301 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:32:47.797374 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.791735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.1.221.129:37053)
0810 22:32:47.792182 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:47.792198 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:47.792227 (+ 29us) server_negotiation.cc:407] Connection header received
0810 22:32:47.792292 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:47.792296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:47.792517 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:47.793129 (+ 612us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:47.793301 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.794138 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.796033 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.796640 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.796763 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.796919 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:47.797098 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 22:32:47.797200 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":129,"threads_started":1}
I20260810 22:32:47.797420 22825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.791518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:37053 (local address 127.0.0.1:60054)
0810 22:32:47.791805 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:47.791819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:47.791896 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:47.792725 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:47.792729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:47.792764 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:47.792943 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.792970 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.794274 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.794281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.795884 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.795895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.796775 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.796778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.796884 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.796922 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 22:32:47.797064 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 22:32:47.797201 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":87,"spinlock_wait_cycles":4352,"thread_start_us":97,"threads_started":1}
I20260810 22:32:47.797435 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.790971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:37823 (local address 127.0.0.1:46874)
0810 22:32:47.791755 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:47.791770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:47.791867 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:47.793014 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:47.793019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:47.793040 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:47.793155 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.793161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.794714 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.794718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.795866 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.795875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.796983 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.796986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.797084 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.797107 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:32:47.797160 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 22:32:47.797248 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":31,"thread_start_us":94,"threads_started":1}
I20260810 22:32:47.797839 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.790759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:43109 (local address 127.0.0.1:50174)
0810 22:32:47.791285 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:47.791324 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:47.791432 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:47.792717 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:47.792723 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:47.792762 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:47.793181 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.793189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.794732 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.794734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.796005 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.796015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.797208 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.797212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.797305 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.797328 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:32:47.797395 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 22:32:47.797696 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":362,"spinlock_wait_cycles":16768,"thread_start_us":78,"threads_started":1}
I20260810 22:32:47.798411 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.791696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.1.221.130:37823)
0810 22:32:47.792537 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:47.792548 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:47.792574 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:32:47.792685 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:47.792690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:47.792855 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:47.793014 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:47.793280 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.794535 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.795990 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.796831 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.797636 (+ 805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.797799 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:47.797850 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 22:32:47.797958 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":98,"threads_started":1}
I20260810 22:32:47.798441 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.790886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.1.221.131:43109)
0810 22:32:47.792147 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:47.792163 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:47.792189 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:32:47.792275 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:47.792280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:47.792512 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:47.792840 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:47.793335 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.794528 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.796116 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.796903 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.796999 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.797148 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:47.798166 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0810 22:32:47.798289 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":123,"threads_started":1}
I20260810 22:32:47.800768 22442 tablet_service.cc:1467] Processing CreateTablet for tablet 499f5a68edcf45f299e06d1f85e219de (DEFAULT_TABLE table=test-table [id=727ccf4042c24e61a843741688120b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:47.801213 22588 tablet_service.cc:1467] Processing CreateTablet for tablet 499f5a68edcf45f299e06d1f85e219de (DEFAULT_TABLE table=test-table [id=727ccf4042c24e61a843741688120b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:47.801803 22442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 499f5a68edcf45f299e06d1f85e219de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:47.801842 22736 tablet_service.cc:1467] Processing CreateTablet for tablet 499f5a68edcf45f299e06d1f85e219de (DEFAULT_TABLE table=test-table [id=727ccf4042c24e61a843741688120b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:47.802295 22588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 499f5a68edcf45f299e06d1f85e219de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:47.802860 22736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 499f5a68edcf45f299e06d1f85e219de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:47.808068 22828 tablet_bootstrap.cc:492] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469: Bootstrap starting.
I20260810 22:32:47.809841 22829 tablet_bootstrap.cc:492] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac: Bootstrap starting.
I20260810 22:32:47.811159 22830 tablet_bootstrap.cc:492] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde: Bootstrap starting.
I20260810 22:32:47.811224 22828 tablet_bootstrap.cc:654] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:47.812314 22828 log.cc:824] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:47.812481 22829 tablet_bootstrap.cc:654] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:47.813113 22830 tablet_bootstrap.cc:654] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:47.813370 22829 log.cc:824] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:47.814368 22830 log.cc:824] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:47.814795 22828 tablet_bootstrap.cc:492] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469: No bootstrap required, opened a new log
I20260810 22:32:47.815009 22828 ts_tablet_manager.cc:1397] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 22:32:47.815500 22829 tablet_bootstrap.cc:492] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac: No bootstrap required, opened a new log
I20260810 22:32:47.815689 22829 ts_tablet_manager.cc:1397] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:32:47.816167 22830 tablet_bootstrap.cc:492] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde: No bootstrap required, opened a new log
I20260810 22:32:47.816358 22830 ts_tablet_manager.cc:1397] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:32:47.819648 22830 raft_consensus.cc:348] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ee890acc45466caea72447b3510fde" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43109 } } peers { permanent_uuid: "63b86e6c722348428ce678b2c8cd86ac" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37823 } } peers { permanent_uuid: "b4244d36d0f1417c970e077340a6b469" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37053 } }
I20260810 22:32:47.819649 22828 raft_consensus.cc:348] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ee890acc45466caea72447b3510fde" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43109 } } peers { permanent_uuid: "63b86e6c722348428ce678b2c8cd86ac" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37823 } } peers { permanent_uuid: "b4244d36d0f1417c970e077340a6b469" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37053 } }
I20260810 22:32:47.819649 22829 raft_consensus.cc:348] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ee890acc45466caea72447b3510fde" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43109 } } peers { permanent_uuid: "63b86e6c722348428ce678b2c8cd86ac" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37823 } } peers { permanent_uuid: "b4244d36d0f1417c970e077340a6b469" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37053 } }
I20260810 22:32:47.819921 22828 raft_consensus.cc:374] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:47.819921 22830 raft_consensus.cc:374] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:47.819921 22829 raft_consensus.cc:374] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:47.819983 22830 raft_consensus.cc:729] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ee890acc45466caea72447b3510fde, State: Initialized, Role: FOLLOWER
I20260810 22:32:47.819983 22828 raft_consensus.cc:729] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4244d36d0f1417c970e077340a6b469, State: Initialized, Role: FOLLOWER
I20260810 22:32:47.819983 22829 raft_consensus.cc:729] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b86e6c722348428ce678b2c8cd86ac, State: Initialized, Role: FOLLOWER
I20260810 22:32:47.820442 22830 consensus_queue.cc:260] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ee890acc45466caea72447b3510fde" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43109 } } peers { permanent_uuid: "63b86e6c722348428ce678b2c8cd86ac" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37823 } } peers { permanent_uuid: "b4244d36d0f1417c970e077340a6b469" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37053 } }
I20260810 22:32:47.820452 22829 consensus_queue.cc:260] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ee890acc45466caea72447b3510fde" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43109 } } peers { permanent_uuid: "63b86e6c722348428ce678b2c8cd86ac" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37823 } } peers { permanent_uuid: "b4244d36d0f1417c970e077340a6b469" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37053 } }
I20260810 22:32:47.820452 22828 consensus_queue.cc:260] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ee890acc45466caea72447b3510fde" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43109 } } peers { permanent_uuid: "63b86e6c722348428ce678b2c8cd86ac" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37823 } } peers { permanent_uuid: "b4244d36d0f1417c970e077340a6b469" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37053 } }
I20260810 22:32:47.821142 22809 heartbeater.cc:503] Master 127.1.221.190:45111 was elected leader, sending a full tablet report...
I20260810 22:32:47.821477 22830 ts_tablet_manager.cc:1428] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 22:32:47.821482 22829 ts_tablet_manager.cc:1428] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:32:47.821482 22828 ts_tablet_manager.cc:1428] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:32:47.821830 22515 heartbeater.cc:503] Master 127.1.221.190:45111 was elected leader, sending a full tablet report...
I20260810 22:32:47.822721 22660 heartbeater.cc:503] Master 127.1.221.190:45111 was elected leader, sending a full tablet report...
I20260810 22:32:47.834385 22836 raft_consensus.cc:482] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:32:47.834599 22836 raft_consensus.cc:504] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ee890acc45466caea72447b3510fde" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43109 } } peers { permanent_uuid: "63b86e6c722348428ce678b2c8cd86ac" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37823 } } peers { permanent_uuid: "b4244d36d0f1417c970e077340a6b469" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37053 } }
I20260810 22:32:47.835702 22836 leader_election.cc:288] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4ee890acc45466caea72447b3510fde (127.1.221.131:43109), b4244d36d0f1417c970e077340a6b469 (127.1.221.129:37053)
I20260810 22:32:47.839399 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.835962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:32889 (local address 127.1.221.131:43109)
0810 22:32:47.836090 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:47.836094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:47.836277 (+ 183us) server_negotiation.cc:407] Connection header received
0810 22:32:47.836337 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:47.836340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:47.836388 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:47.836472 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:47.836747 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.837259 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.838408 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.838828 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.838966 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.839018 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:47.839205 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 22:32:47.839277 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:32:47.839399 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.835921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:43109 (local address 127.1.221.130:32889)
0810 22:32:47.836128 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:47.836146 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:47.836242 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:47.836483 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:47.836486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:47.836504 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:47.836633 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.836639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.837372 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.837378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.838275 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.838285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.838958 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.838961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.839075 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.839095 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 22:32:47.839186 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 22:32:47.839270 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":7}
I20260810 22:32:47.839560 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.836094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:59611 (local address 127.1.221.129:37053)
0810 22:32:47.836247 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:47.836252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:47.836449 (+ 197us) server_negotiation.cc:407] Connection header received
0810 22:32:47.836573 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:47.836577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:47.836631 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:47.836716 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:47.836994 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.837499 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.838573 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.838969 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.839096 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.839146 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:47.839343 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 22:32:47.839406 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:32:47.839792 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.835922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:37053 (local address 127.1.221.130:59611)
0810 22:32:47.836357 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:47.836368 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:47.836486 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:47.836733 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:47.836736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:47.836751 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:47.836880 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.836885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.837642 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.837645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.838445 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:47.838454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.839092 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.839096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:47.839199 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.839229 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 22:32:47.839325 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 22:32:47.839395 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":63,"thread_start_us":143,"threads_started":1}
I20260810 22:32:47.840101 22759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499f5a68edcf45f299e06d1f85e219de" candidate_uuid: "63b86e6c722348428ce678b2c8cd86ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4ee890acc45466caea72447b3510fde" is_pre_election: true
I20260810 22:32:47.840387 22463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499f5a68edcf45f299e06d1f85e219de" candidate_uuid: "63b86e6c722348428ce678b2c8cd86ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4244d36d0f1417c970e077340a6b469" is_pre_election: true
I20260810 22:32:47.840444 22759 raft_consensus.cc:2454] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b86e6c722348428ce678b2c8cd86ac in term 0.
I20260810 22:32:47.840667 22463 raft_consensus.cc:2454] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b86e6c722348428ce678b2c8cd86ac in term 0.
I20260810 22:32:47.841024 22538 leader_election.cc:302] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b86e6c722348428ce678b2c8cd86ac, d4ee890acc45466caea72447b3510fde; no voters:
I20260810 22:32:47.841323 22836 raft_consensus.cc:2781] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:32:47.841415 22836 raft_consensus.cc:482] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:32:47.841459 22836 raft_consensus.cc:3037] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:47.842571 22836 raft_consensus.cc:504] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ee890acc45466caea72447b3510fde" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43109 } } peers { permanent_uuid: "63b86e6c722348428ce678b2c8cd86ac" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37823 } } peers { permanent_uuid: "b4244d36d0f1417c970e077340a6b469" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37053 } }
I20260810 22:32:47.842957 22836 leader_election.cc:288] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [CANDIDATE]: Term 1 election: Requested vote from peers d4ee890acc45466caea72447b3510fde (127.1.221.131:43109), b4244d36d0f1417c970e077340a6b469 (127.1.221.129:37053)
I20260810 22:32:47.843413 22463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499f5a68edcf45f299e06d1f85e219de" candidate_uuid: "63b86e6c722348428ce678b2c8cd86ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4244d36d0f1417c970e077340a6b469"
I20260810 22:32:47.843559 22463 raft_consensus.cc:3037] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:47.844403 22759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "499f5a68edcf45f299e06d1f85e219de" candidate_uuid: "63b86e6c722348428ce678b2c8cd86ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4ee890acc45466caea72447b3510fde"
I20260810 22:32:47.844535 22759 raft_consensus.cc:3037] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:47.844918 22463 raft_consensus.cc:2454] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b86e6c722348428ce678b2c8cd86ac in term 1.
I20260810 22:32:47.845283 22541 leader_election.cc:302] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b86e6c722348428ce678b2c8cd86ac, b4244d36d0f1417c970e077340a6b469; no voters:
I20260810 22:32:47.845458 22836 raft_consensus.cc:2781] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:47.845726 22836 raft_consensus.cc:686] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [term 1 LEADER]: Becoming Leader. State: Replica: 63b86e6c722348428ce678b2c8cd86ac, State: Running, Role: LEADER
I20260810 22:32:47.845813 22759 raft_consensus.cc:2454] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b86e6c722348428ce678b2c8cd86ac in term 1.
I20260810 22:32:47.846081 22836 consensus_queue.cc:237] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ee890acc45466caea72447b3510fde" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43109 } } peers { permanent_uuid: "63b86e6c722348428ce678b2c8cd86ac" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37823 } } peers { permanent_uuid: "b4244d36d0f1417c970e077340a6b469" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37053 } }
I20260810 22:32:47.849259 22298 catalog_manager.cc:5495] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 63b86e6c722348428ce678b2c8cd86ac (127.1.221.130). New cstate: current_term: 1 leader_uuid: "63b86e6c722348428ce678b2c8cd86ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ee890acc45466caea72447b3510fde" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b86e6c722348428ce678b2c8cd86ac" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 37823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4244d36d0f1417c970e077340a6b469" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37053 } health_report { overall_health: UNKNOWN } } }
I20260810 22:32:47.867496 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.863719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.1.221.130:37823)
0810 22:32:47.863861 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:47.863865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:47.863878 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:32:47.863995 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:47.863997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:47.864068 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:47.864150 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:32:47.864385 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.864944 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.865905 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:47.866125 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:47.866243 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:47.866618 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:32:47.867018 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:32:47.867184 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:47.867281 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 22:32:47.867322 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:32:47.873533 22759 raft_consensus.cc:1261] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde [term 1 FOLLOWER]: Refusing update from remote peer 63b86e6c722348428ce678b2c8cd86ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:47.873533 22463 raft_consensus.cc:1261] T 499f5a68edcf45f299e06d1f85e219de P b4244d36d0f1417c970e077340a6b469 [term 1 FOLLOWER]: Refusing update from remote peer 63b86e6c722348428ce678b2c8cd86ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:47.874343 22839 consensus_queue.cc:1035] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4244d36d0f1417c970e077340a6b469" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:32:47.874527 22836 consensus_queue.cc:1035] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4ee890acc45466caea72447b3510fde" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 43109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:32:47.881876 22842 mvcc.cc:205] Tried to move back new op lower bound from 7317099183602503680 to 7317099183500529664. Current Snapshot: MvccSnapshot[applied={T|T < 7317099183602503680}]
I20260810 22:32:47.882494 22844 mvcc.cc:205] Tried to move back new op lower bound from 7317099183602503680 to 7317099183500529664. Current Snapshot: MvccSnapshot[applied={T|T < 7317099183602503680}]
I20260810 22:32:47.891122 22298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:47.891309 22298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:47.893712 22298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c998d6b1a3d457b885e97deb1326ced: Sending DeleteTablet for 3 replicas of tablet 499f5a68edcf45f299e06d1f85e219de
I20260810 22:32:47.894604 22736 tablet_service.cc:1514] Processing DeleteTablet for tablet 499f5a68edcf45f299e06d1f85e219de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:47 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:50174
I20260810 22:32:47.895066 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 22369
I20260810 22:32:47.897455 22588 tablet_service.cc:1514] Processing DeleteTablet for tablet 499f5a68edcf45f299e06d1f85e219de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:47 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:46874
I20260810 22:32:47.899080 22854 tablet_replica.cc:330] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde: stopping tablet replica
I20260810 22:32:47.899398 22854 raft_consensus.cc:2229] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:32:47.899713 22854 raft_consensus.cc:2258] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:47.901193 22854 ts_tablet_manager.cc:1905] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:47.903422 22854 ts_tablet_manager.cc:1918] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:47.903538 22854 log.cc:1196] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/499f5a68edcf45f299e06d1f85e219de
I20260810 22:32:47.903856 22854 ts_tablet_manager.cc:1939] T 499f5a68edcf45f299e06d1f85e219de P d4ee890acc45466caea72447b3510fde: Deleting consensus metadata
I20260810 22:32:47.904947 22283 catalog_manager.cc:4841] TS d4ee890acc45466caea72447b3510fde (127.1.221.131:43109): tablet 499f5a68edcf45f299e06d1f85e219de (table test-table [id=727ccf4042c24e61a843741688120b79]) successfully deleted
I20260810 22:32:47.905789 22855 tablet_replica.cc:330] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac: stopping tablet replica
I20260810 22:32:47.906090 22855 raft_consensus.cc:2229] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:32:47.906459 22855 raft_consensus.cc:2258] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:47.907953 22855 ts_tablet_manager.cc:1905] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:47.910357 22855 ts_tablet_manager.cc:1918] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:47.910478 22855 log.cc:1196] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/499f5a68edcf45f299e06d1f85e219de
I20260810 22:32:47.910802 22855 ts_tablet_manager.cc:1939] T 499f5a68edcf45f299e06d1f85e219de P 63b86e6c722348428ce678b2c8cd86ac: Deleting consensus metadata
I20260810 22:32:47.912034 22287 catalog_manager.cc:4841] TS 63b86e6c722348428ce678b2c8cd86ac (127.1.221.130:37823): tablet 499f5a68edcf45f299e06d1f85e219de (table test-table [id=727ccf4042c24e61a843741688120b79]) successfully deleted
W20260810 22:32:47.918610 22287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:37053 (error 108)
I20260810 22:32:47.918840 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 22520
I20260810 22:32:47.922077 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:47.921561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:37053 (local address 127.0.0.1:60068)
0810 22:32:47.921873 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:47.921963 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
W20260810 22:32:47.922621 22287 catalog_manager.cc:4584] TS b4244d36d0f1417c970e077340a6b469 (127.1.221.129:37053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 499f5a68edcf45f299e06d1f85e219de: Network error: Client connection negotiation failed: client connection to 127.1.221.129:37053: connect: Connection refused (error 111)
I20260810 22:32:47.927789 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 22665
I20260810 22:32:47.935222 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 22264
2026-08-10T22:32:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2372 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T22:32:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:32:47Z Disabled control of system clock
I20260810 22:32:47.969517 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40069
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 22:32:48.081013 22860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:48.081278 22860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:48.081313 22860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:48.085328 22860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:32:48.085387 22860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:48.085403 22860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:48.085420 22860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:32:48.085434 22860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:32:48.090142 22860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:48.091409 22860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:48.092756 22860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:32:48.101399 22860 server_base.cc:1034] running on GCE node
W20260810 22:32:48.101336 22866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:48.101346 22865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:48.101365 22868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:48.102146 22860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:48.103147 22860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:48.104303 22860 hybrid_clock.cc:648] HybridClock initialized: now 1786401168104286 us; error 29 us; skew 500 ppm
I20260810 22:32:48.106164 22860 webserver.cc:460] Webserver started at http://127.1.221.190:35899/ using document root <none> and password file <none>
I20260810 22:32:48.106691 22860 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:48.106753 22860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:48.106917 22860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:48.108894 22860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "5bdba4d231da44118d82ceb1a61e781e"
format_stamp: "Formatted at 2026-08-10 22:32:48 on dist-test-slave-njxd"
server_key: "1a340f7e78b56fac5a0f68e0606a90e5"
server_key_iv: "d434f60fb813f1a56a452698a14eddfa"
server_key_version: "encryptionkey@0"
I20260810 22:32:48.109488 22860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "5bdba4d231da44118d82ceb1a61e781e"
format_stamp: "Formatted at 2026-08-10 22:32:48 on dist-test-slave-njxd"
server_key: "1a340f7e78b56fac5a0f68e0606a90e5"
server_key_iv: "d434f60fb813f1a56a452698a14eddfa"
server_key_version: "encryptionkey@0"
I20260810 22:32:48.113107 22860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:32:48.115533 22874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:48.116765 22860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:48.116900 22860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "5bdba4d231da44118d82ceb1a61e781e"
format_stamp: "Formatted at 2026-08-10 22:32:48 on dist-test-slave-njxd"
server_key: "1a340f7e78b56fac5a0f68e0606a90e5"
server_key_iv: "d434f60fb813f1a56a452698a14eddfa"
server_key_version: "encryptionkey@0"
I20260810 22:32:48.117010 22860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:48.142836 22860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:48.143616 22860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:48.143833 22860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:48.151101 22860 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:40069
I20260810 22:32:48.151114 22935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:40069 every 8 connection(s)
I20260810 22:32:48.152072 22860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:32:48.155364 22936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:48.155905 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 22860
I20260810 22:32:48.156036 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:32:48.156381 1910 external_mini_cluster.cc:1428] Setting key 301e2554529f4586702542ca4a40bacf
I20260810 22:32:48.161891 22936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e: Bootstrap starting.
I20260810 22:32:48.164425 22936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:48.165418 22936 log.cc:824] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:48.166476 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.157943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.1.221.190:40069)
0810 22:32:48.158606 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:48.158621 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:48.158657 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:32:48.159327 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:48.159344 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:48.159837 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:48.160178 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:48.160656 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.161678 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.162648 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.163184 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.163353 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.164238 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:48.164244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:48.164262 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:48.164885 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:48.164972 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:48.164999 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:48.165115 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:48.165256 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 22:32:48.165539 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":138,"threads_started":1}
I20260810 22:32:48.167642 22936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e: No bootstrap required, opened a new log
I20260810 22:32:48.171422 22936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdba4d231da44118d82ceb1a61e781e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40069 } }
I20260810 22:32:48.171722 22936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:48.171802 22936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bdba4d231da44118d82ceb1a61e781e, State: Initialized, Role: FOLLOWER
I20260810 22:32:48.172286 22936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdba4d231da44118d82ceb1a61e781e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40069 } }
I20260810 22:32:48.172429 22936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:32:48.172492 22936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:32:48.172581 22936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:48.173499 22936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdba4d231da44118d82ceb1a61e781e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40069 } }
I20260810 22:32:48.173870 22936 leader_election.cc:302] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bdba4d231da44118d82ceb1a61e781e; no voters:
I20260810 22:32:48.174201 22936 leader_election.cc:288] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:32:48.174378 22941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:48.174651 22941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [term 1 LEADER]: Becoming Leader. State: Replica: 5bdba4d231da44118d82ceb1a61e781e, State: Running, Role: LEADER
I20260810 22:32:48.174971 22941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdba4d231da44118d82ceb1a61e781e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40069 } }
I20260810 22:32:48.175524 22936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:32:48.176746 22943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bdba4d231da44118d82ceb1a61e781e. Latest consensus state: current_term: 1 leader_uuid: "5bdba4d231da44118d82ceb1a61e781e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdba4d231da44118d82ceb1a61e781e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40069 } } }
I20260810 22:32:48.176743 22942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bdba4d231da44118d82ceb1a61e781e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bdba4d231da44118d82ceb1a61e781e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40069 } } }
I20260810 22:32:48.176909 22942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:48.176908 22943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:48.177349 22950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:32:48.180662 22950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:32:48.186561 22950 catalog_manager.cc:1347] Generated new cluster ID: 6a0214313b614ad6b6794e2672bc9e4c
I20260810 22:32:48.186646 22950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:32:48.214974 22950 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:32:48.215962 22950 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:32:48.224153 22950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e: Generated new TSK 0
I20260810 22:32:48.224877 22950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:32:48.236779 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40069
--builtin_ntp_servers=127.1.221.148:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 22:32:48.369199 22960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:48.369465 22960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:48.369505 22960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:48.373898 22960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:48.373967 22960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:48.374025 22960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:32:48.379029 22960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:48.380142 22960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:48.381506 22960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:48.391301 22965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:48.391382 22968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:48.391305 22970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:48.391546 22960 server_base.cc:1034] running on GCE node
I20260810 22:32:48.392058 22960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:48.392707 22960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:48.393893 22960 hybrid_clock.cc:648] HybridClock initialized: now 1786401168393879 us; error 26 us; skew 500 ppm
I20260810 22:32:48.395777 22960 webserver.cc:460] Webserver started at http://127.1.221.129:39775/ using document root <none> and password file <none>
I20260810 22:32:48.396366 22960 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:48.396458 22960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:48.396697 22960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:48.398766 22960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "39fa4a4204d245adbe2573d32194003f"
format_stamp: "Formatted at 2026-08-10 22:32:48 on dist-test-slave-njxd"
server_key: "ce20968dc87bd3ee0be504faaea5bc8b"
server_key_iv: "018712e390e39c629247e38265d916d7"
server_key_version: "encryptionkey@0"
I20260810 22:32:48.399328 22960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "39fa4a4204d245adbe2573d32194003f"
format_stamp: "Formatted at 2026-08-10 22:32:48 on dist-test-slave-njxd"
server_key: "ce20968dc87bd3ee0be504faaea5bc8b"
server_key_iv: "018712e390e39c629247e38265d916d7"
server_key_version: "encryptionkey@0"
I20260810 22:32:48.403069 22960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:32:48.405617 22978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:48.406859 22960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:32:48.406997 22960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "39fa4a4204d245adbe2573d32194003f"
format_stamp: "Formatted at 2026-08-10 22:32:48 on dist-test-slave-njxd"
server_key: "ce20968dc87bd3ee0be504faaea5bc8b"
server_key_iv: "018712e390e39c629247e38265d916d7"
server_key_version: "encryptionkey@0"
I20260810 22:32:48.407112 22960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:48.425091 22960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:48.426127 22960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:48.426368 22960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:48.427150 22960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:48.428558 22960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:48.428634 22960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:48.428694 22960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:48.428733 22960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:48.439553 22960 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:43749
I20260810 22:32:48.439574 23104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:43749 every 8 connection(s)
I20260810 22:32:48.440528 22960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:32:48.445647 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 22960
I20260810 22:32:48.445792 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:32:48.446090 1910 external_mini_cluster.cc:1428] Setting key e40abca7e251f9c421cf2ed0848f96a1
I20260810 22:32:48.448056 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40069
--builtin_ntp_servers=127.1.221.148:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 22:32:48.450836 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.442595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:54807 (local address 127.1.221.190:40069)
0810 22:32:48.442737 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:48.442740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:48.443804 (+ 1064us) server_negotiation.cc:407] Connection header received
0810 22:32:48.445260 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:48.445264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:48.445324 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:48.445406 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:48.446521 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.447166 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.448533 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.448936 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.449059 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.450320 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:48.450323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:48.450325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:48.450337 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:48.450358 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:48.450366 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:48.450428 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:48.450661 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 22:32:48.450722 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:32:48.452095 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.442965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40069 (local address 127.1.221.129:54807)
0810 22:32:48.443649 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:48.443692 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:48.445038 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:48.445640 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:48.445652 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:48.445780 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:48.446376 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.446390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.447348 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.447352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.448404 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.448415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.449089 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.449092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.449241 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.449985 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:32:48.449988 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:48.450055 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:32:48.450149 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:48.450457 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:48.450468 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:48.450476 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:32:48.450615 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 22:32:48.450960 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":130,"threads_started":1}
I20260810 22:32:48.453460 23105 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40069
I20260810 22:32:48.453769 23105 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:48.454326 23105 heartbeater.cc:511] Master 127.1.221.190:40069 requested a full tablet report, sending...
I20260810 22:32:48.455894 22894 ts_manager.cc:194] Registered new tserver with Master: 39fa4a4204d245adbe2573d32194003f (127.1.221.129:43749)
I20260810 22:32:48.457064 22894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.129:54807
W20260810 22:32:48.563899 23108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:48.564213 23108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:48.564294 23108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:48.568288 23108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:48.568378 23108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:48.568470 23108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:32:48.573153 23108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:48.574324 23108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:48.575847 23108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:48.585882 23114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:48.585896 23113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:48.586148 23108 server_base.cc:1034] running on GCE node
W20260810 22:32:48.585896 23117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:48.586608 23108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:48.587248 23108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:48.588472 23108 hybrid_clock.cc:648] HybridClock initialized: now 1786401168588457 us; error 27 us; skew 500 ppm
I20260810 22:32:48.590312 23108 webserver.cc:460] Webserver started at http://127.1.221.130:35015/ using document root <none> and password file <none>
I20260810 22:32:48.590873 23108 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:48.590936 23108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:48.591182 23108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:48.593287 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "b36d290962464d1b8135813b736c3d3c"
format_stamp: "Formatted at 2026-08-10 22:32:48 on dist-test-slave-njxd"
server_key: "2803b1a32814a9ff45eda90b101622ea"
server_key_iv: "bb63c7ab200e15c742568dc67653f405"
server_key_version: "encryptionkey@0"
I20260810 22:32:48.593852 23108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "b36d290962464d1b8135813b736c3d3c"
format_stamp: "Formatted at 2026-08-10 22:32:48 on dist-test-slave-njxd"
server_key: "2803b1a32814a9ff45eda90b101622ea"
server_key_iv: "bb63c7ab200e15c742568dc67653f405"
server_key_version: "encryptionkey@0"
I20260810 22:32:48.597343 23108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:32:48.599759 23123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:48.600993 23108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:48.601083 23108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "b36d290962464d1b8135813b736c3d3c"
format_stamp: "Formatted at 2026-08-10 22:32:48 on dist-test-slave-njxd"
server_key: "2803b1a32814a9ff45eda90b101622ea"
server_key_iv: "bb63c7ab200e15c742568dc67653f405"
server_key_version: "encryptionkey@0"
I20260810 22:32:48.601163 23108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:48.613729 23108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:48.614354 23108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:48.614503 23108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:48.615036 23108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:48.615972 23108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:48.616048 23108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:48.616117 23108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:48.616163 23108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:48.625265 23108 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:44495
I20260810 22:32:48.625288 23249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:44495 every 8 connection(s)
I20260810 22:32:48.626291 23108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:32:48.635528 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.628133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:43637 (local address 127.1.221.190:40069)
0810 22:32:48.628276 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:48.628279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:48.629309 (+ 1030us) server_negotiation.cc:407] Connection header received
0810 22:32:48.630789 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:48.630793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:48.630852 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:48.630960 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:48.631873 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.632402 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.633547 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.633833 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.633949 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.635032 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:48.635035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:48.635037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:48.635048 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:48.635067 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:48.635074 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:48.635158 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:48.635317 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 22:32:48.635380 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:32:48.636521 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.628498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40069 (local address 127.1.221.130:43637)
0810 22:32:48.629156 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:48.629197 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:48.630546 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:48.631142 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:48.631149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:48.631236 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:48.631722 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.631734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.632531 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.632536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.633413 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.633423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.633993 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.633997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.634165 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.634760 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:32:48.634761 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:48.634801 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:32:48.634864 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:48.635160 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:48.635172 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:48.635181 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 22:32:48.635317 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 22:32:48.635616 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":123,"threads_started":1}
I20260810 22:32:48.636567 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 23108
I20260810 22:32:48.636704 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:32:48.637061 1910 external_mini_cluster.cc:1428] Setting key 02299b89023e83d56fc783213a3c08c0
I20260810 22:32:48.637823 23250 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40069
I20260810 22:32:48.638079 23250 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:48.638728 23250 heartbeater.cc:511] Master 127.1.221.190:40069 requested a full tablet report, sending...
I20260810 22:32:48.639314 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40069
--builtin_ntp_servers=127.1.221.148:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 22:32:48.639863 22894 ts_manager.cc:194] Registered new tserver with Master: b36d290962464d1b8135813b736c3d3c (127.1.221.130:44495)
I20260810 22:32:48.640451 22894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.130:43637
W20260810 22:32:48.753516 23253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:48.753852 23253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:48.753894 23253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:48.758167 23253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:48.758255 23253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:48.758313 23253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:32:48.763054 23253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:48.764145 23253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:48.765481 23253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:48.775054 23262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:48.775228 23259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:48.775269 23260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:48.775496 23253 server_base.cc:1034] running on GCE node
I20260810 22:32:48.775966 23253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:48.776607 23253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:48.777803 23253 hybrid_clock.cc:648] HybridClock initialized: now 1786401168777789 us; error 26 us; skew 500 ppm
I20260810 22:32:48.779711 23253 webserver.cc:460] Webserver started at http://127.1.221.131:43213/ using document root <none> and password file <none>
I20260810 22:32:48.780303 23253 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:48.780397 23253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:48.780637 23253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:48.782748 23253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "4eec9ff2f2684f0abc1a7d25867c0952"
format_stamp: "Formatted at 2026-08-10 22:32:48 on dist-test-slave-njxd"
server_key: "694d51289c0d58cf67ff1fd5aaf48fdd"
server_key_iv: "187bfe4dadf02e9e24d32759d7037dab"
server_key_version: "encryptionkey@0"
I20260810 22:32:48.783325 23253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "4eec9ff2f2684f0abc1a7d25867c0952"
format_stamp: "Formatted at 2026-08-10 22:32:48 on dist-test-slave-njxd"
server_key: "694d51289c0d58cf67ff1fd5aaf48fdd"
server_key_iv: "187bfe4dadf02e9e24d32759d7037dab"
server_key_version: "encryptionkey@0"
I20260810 22:32:48.786995 23253 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:32:48.789412 23268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:48.790634 23253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:48.790768 23253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "4eec9ff2f2684f0abc1a7d25867c0952"
format_stamp: "Formatted at 2026-08-10 22:32:48 on dist-test-slave-njxd"
server_key: "694d51289c0d58cf67ff1fd5aaf48fdd"
server_key_iv: "187bfe4dadf02e9e24d32759d7037dab"
server_key_version: "encryptionkey@0"
I20260810 22:32:48.790884 23253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:48.803680 23253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:48.804375 23253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:48.804567 23253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:48.805205 23253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:48.806200 23253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:48.806275 23253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:48.806347 23253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:48.806382 23253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:48.815596 23253 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:37591
I20260810 22:32:48.815634 23392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:37591 every 8 connection(s)
I20260810 22:32:48.816532 23253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:32:48.825984 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.818360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:52885 (local address 127.1.221.190:40069)
0810 22:32:48.818498 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:48.818502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:48.819573 (+ 1071us) server_negotiation.cc:407] Connection header received
0810 22:32:48.821268 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:48.821271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:48.821331 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:48.821438 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:48.822407 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.822928 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.824081 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.824305 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.824417 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.825474 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:48.825477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:48.825479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:48.825489 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:48.825507 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:48.825514 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:48.825607 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:48.825789 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 22:32:48.825848 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:32:48.826583 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 23253
I20260810 22:32:48.826710 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:32:48.827025 1910 external_mini_cluster.cc:1428] Setting key 43677b02b62772e54dd535ff80dea5f7
I20260810 22:32:48.827059 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.818745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40069 (local address 127.1.221.131:52885)
0810 22:32:48.819383 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:48.819423 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:48.821049 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:48.821675 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:48.821683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:48.821766 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:48.822259 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.822271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.823067 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.823071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.823943 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.823953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.824462 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.824466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.824630 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.825198 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:32:48.825200 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:48.825238 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:32:48.825297 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:48.825644 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:48.825653 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:48.825658 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 22:32:48.825788 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 22:32:48.826051 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20260810 22:32:48.828320 23393 heartbeater.cc:347] Connected to a master server at 127.1.221.190:40069
I20260810 22:32:48.828547 23393 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:48.829161 23393 heartbeater.cc:511] Master 127.1.221.190:40069 requested a full tablet report, sending...
I20260810 22:32:48.830159 22894 ts_manager.cc:194] Registered new tserver with Master: 4eec9ff2f2684f0abc1a7d25867c0952 (127.1.221.131:37591)
I20260810 22:32:48.830670 22894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.131:52885
I20260810 22:32:48.841027 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:32:48.846568 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.842969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56014 (local address 127.1.221.190:40069)
0810 22:32:48.843126 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:48.843130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:48.843210 (+ 80us) server_negotiation.cc:407] Connection header received
0810 22:32:48.843310 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:48.843312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:48.843351 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:48.843414 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:48.843673 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.844182 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.844944 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.845152 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.845222 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.846056 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:48.846059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:48.846061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:48.846072 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:48.846089 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:48.846097 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:48.846195 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:48.846331 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 22:32:48.846399 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 22:32:48.849251 22894 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:32:48.852332 22894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:32:48.867352 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.861273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.1.221.130:44495)
0810 22:32:48.862108 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:48.862121 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:48.862150 (+ 29us) server_negotiation.cc:407] Connection header received
0810 22:32:48.862476 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:48.862484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:48.862710 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:48.862857 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:48.863428 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.864635 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.866032 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.866652 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.866774 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.866987 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:48.867078 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 22:32:48.867182 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":103,"threads_started":1}
I20260810 22:32:48.867350 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.861131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:44495 (local address 127.0.0.1:56632)
0810 22:32:48.861523 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:48.861569 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:48.862251 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:48.863072 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:48.863078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:48.863099 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:48.863290 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.863299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.864785 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.864789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.865897 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.865909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.866784 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.866788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.866895 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.866933 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 22:32:48.867058 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 22:32:48.867187 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":26,"spinlock_wait_cycles":13056,"thread_start_us":123,"threads_started":1}
I20260810 22:32:48.867529 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.861152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:37591 (local address 127.0.0.1:47250)
0810 22:32:48.862313 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:48.862328 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:48.862420 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:48.863106 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:48.863109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:48.863128 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:48.863295 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.863302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.864337 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.864344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.865805 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.865816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.867099 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.867103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.867198 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.867219 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 22:32:48.867307 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 22:32:48.867374 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"mutex_wait_us":83,"thread_start_us":105,"threads_started":1}
I20260810 22:32:48.867883 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.862106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.1.221.131:37591)
0810 22:32:48.862728 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:48.862741 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:48.862769 (+ 28us) server_negotiation.cc:407] Connection header received
0810 22:32:48.862830 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:48.862833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:48.862970 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:48.863131 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:48.863432 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.864216 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.865939 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.866735 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.866846 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.867035 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:48.867624 (+ 589us) server_negotiation.cc:299] Negotiation successful
0810 22:32:48.867725 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":168,"threads_started":1}
I20260810 22:32:48.867944 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.861771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43749 (local address 127.0.0.1:57066)
0810 22:32:48.862552 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:48.862567 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:48.862655 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:48.863292 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:48.863295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:48.863311 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:48.863401 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.863408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.864815 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.864818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.866055 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.866064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.867279 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.867285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.867366 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.867385 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:32:48.867448 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 22:32:48.867822 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":51,"thread_start_us":95,"threads_started":1}
I20260810 22:32:48.868573 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.861948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57066 (local address 127.1.221.129:43749)
0810 22:32:48.862822 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:48.862838 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:48.862861 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:32:48.862935 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:48.862940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:48.863129 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:48.863286 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:48.863494 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.864651 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.866187 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.866952 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.867059 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.867217 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:48.868155 (+ 938us) server_negotiation.cc:299] Negotiation successful
0810 22:32:48.868407 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":112,"threads_started":1}
I20260810 22:32:48.870623 23177 tablet_service.cc:1467] Processing CreateTablet for tablet d69f4d65a4b64f26bc522c1df0ee745c (DEFAULT_TABLE table=test-table [id=88cdfd4e931347fb82ac878aeb433899]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:48.871141 23319 tablet_service.cc:1467] Processing CreateTablet for tablet d69f4d65a4b64f26bc522c1df0ee745c (DEFAULT_TABLE table=test-table [id=88cdfd4e931347fb82ac878aeb433899]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:48.871141 23031 tablet_service.cc:1467] Processing CreateTablet for tablet d69f4d65a4b64f26bc522c1df0ee745c (DEFAULT_TABLE table=test-table [id=88cdfd4e931347fb82ac878aeb433899]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:48.871587 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d69f4d65a4b64f26bc522c1df0ee745c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:48.872135 23031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d69f4d65a4b64f26bc522c1df0ee745c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:48.872135 23319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d69f4d65a4b64f26bc522c1df0ee745c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:48.878384 23410 tablet_bootstrap.cc:492] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c: Bootstrap starting.
I20260810 22:32:48.879942 23411 tablet_bootstrap.cc:492] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f: Bootstrap starting.
I20260810 22:32:48.880506 23410 tablet_bootstrap.cc:654] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:48.881192 23410 log.cc:824] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:48.881688 23411 tablet_bootstrap.cc:654] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:48.881919 23412 tablet_bootstrap.cc:492] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952: Bootstrap starting.
I20260810 22:32:48.882519 23411 log.cc:824] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:48.883343 23410 tablet_bootstrap.cc:492] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c: No bootstrap required, opened a new log
I20260810 22:32:48.883527 23410 ts_tablet_manager.cc:1397] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:32:48.883873 23412 tablet_bootstrap.cc:654] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:48.884454 23411 tablet_bootstrap.cc:492] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f: No bootstrap required, opened a new log
I20260810 22:32:48.884599 23412 log.cc:824] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:48.884627 23411 ts_tablet_manager.cc:1397] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 22:32:48.886174 23412 tablet_bootstrap.cc:492] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952: No bootstrap required, opened a new log
I20260810 22:32:48.886370 23412 ts_tablet_manager.cc:1397] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 22:32:48.887840 23410 raft_consensus.cc:348] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d290962464d1b8135813b736c3d3c" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 44495 } } peers { permanent_uuid: "4eec9ff2f2684f0abc1a7d25867c0952" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37591 } } peers { permanent_uuid: "39fa4a4204d245adbe2573d32194003f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43749 } }
I20260810 22:32:48.888091 23410 raft_consensus.cc:374] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:48.888144 23410 raft_consensus.cc:729] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36d290962464d1b8135813b736c3d3c, State: Initialized, Role: FOLLOWER
I20260810 22:32:48.888603 23410 consensus_queue.cc:260] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d290962464d1b8135813b736c3d3c" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 44495 } } peers { permanent_uuid: "4eec9ff2f2684f0abc1a7d25867c0952" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37591 } } peers { permanent_uuid: "39fa4a4204d245adbe2573d32194003f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43749 } }
I20260810 22:32:48.888674 23411 raft_consensus.cc:348] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d290962464d1b8135813b736c3d3c" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 44495 } } peers { permanent_uuid: "4eec9ff2f2684f0abc1a7d25867c0952" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37591 } } peers { permanent_uuid: "39fa4a4204d245adbe2573d32194003f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43749 } }
I20260810 22:32:48.888935 23411 raft_consensus.cc:374] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:48.889017 23411 raft_consensus.cc:729] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39fa4a4204d245adbe2573d32194003f, State: Initialized, Role: FOLLOWER
I20260810 22:32:48.889436 23411 consensus_queue.cc:260] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d290962464d1b8135813b736c3d3c" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 44495 } } peers { permanent_uuid: "4eec9ff2f2684f0abc1a7d25867c0952" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37591 } } peers { permanent_uuid: "39fa4a4204d245adbe2573d32194003f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43749 } }
I20260810 22:32:48.889660 23410 ts_tablet_manager.cc:1428] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:32:48.889987 23250 heartbeater.cc:503] Master 127.1.221.190:40069 was elected leader, sending a full tablet report...
I20260810 22:32:48.889853 23412 raft_consensus.cc:348] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d290962464d1b8135813b736c3d3c" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 44495 } } peers { permanent_uuid: "4eec9ff2f2684f0abc1a7d25867c0952" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37591 } } peers { permanent_uuid: "39fa4a4204d245adbe2573d32194003f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43749 } }
I20260810 22:32:48.890115 23412 raft_consensus.cc:374] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:48.890170 23412 raft_consensus.cc:729] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eec9ff2f2684f0abc1a7d25867c0952, State: Initialized, Role: FOLLOWER
I20260810 22:32:48.890329 23411 ts_tablet_manager.cc:1428] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:32:48.890579 23412 consensus_queue.cc:260] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d290962464d1b8135813b736c3d3c" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 44495 } } peers { permanent_uuid: "4eec9ff2f2684f0abc1a7d25867c0952" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37591 } } peers { permanent_uuid: "39fa4a4204d245adbe2573d32194003f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43749 } }
I20260810 22:32:48.890663 23105 heartbeater.cc:503] Master 127.1.221.190:40069 was elected leader, sending a full tablet report...
I20260810 22:32:48.891453 23412 ts_tablet_manager.cc:1428] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 22:32:48.891512 23393 heartbeater.cc:503] Master 127.1.221.190:40069 was elected leader, sending a full tablet report...
I20260810 22:32:48.900838 23418 raft_consensus.cc:482] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:32:48.900986 23418 raft_consensus.cc:504] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d290962464d1b8135813b736c3d3c" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 44495 } } peers { permanent_uuid: "4eec9ff2f2684f0abc1a7d25867c0952" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37591 } } peers { permanent_uuid: "39fa4a4204d245adbe2573d32194003f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43749 } }
I20260810 22:32:48.902087 23418 leader_election.cc:288] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eec9ff2f2684f0abc1a7d25867c0952 (127.1.221.131:37591), 39fa4a4204d245adbe2573d32194003f (127.1.221.129:43749)
I20260810 22:32:48.905980 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.902366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:35867 (local address 127.1.221.131:37591)
0810 22:32:48.902491 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:48.902494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:48.902507 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:32:48.902610 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:48.902613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:48.902650 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:48.902725 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:48.903045 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.903761 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.904818 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.905399 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.905518 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.905611 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:48.905816 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 22:32:48.905885 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:32:48.905987 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.902285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:37591 (local address 127.1.221.130:35867)
0810 22:32:48.902402 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:48.902418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:48.902502 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:48.902782 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:48.902785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:48.902805 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:48.902926 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.902932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.903869 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.903875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.904655 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.904665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.905511 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.905514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.905663 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.905691 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 22:32:48.905787 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 22:32:48.905860 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 22:32:48.906649 23342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69f4d65a4b64f26bc522c1df0ee745c" candidate_uuid: "b36d290962464d1b8135813b736c3d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec9ff2f2684f0abc1a7d25867c0952" is_pre_election: true
I20260810 22:32:48.906949 23342 raft_consensus.cc:2454] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b36d290962464d1b8135813b736c3d3c in term 0.
I20260810 22:32:48.907466 23124 leader_election.cc:302] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eec9ff2f2684f0abc1a7d25867c0952, b36d290962464d1b8135813b736c3d3c; no voters:
I20260810 22:32:48.907694 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.902298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43749 (local address 127.1.221.130:34249)
0810 22:32:48.902958 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:48.902970 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:48.903089 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:48.903293 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:48.903296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:48.903313 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:48.903431 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.903436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.904282 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.904285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.905491 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:48.905501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.907282 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.907289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:48.907389 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.907412 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:32:48.907499 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 22:32:48.907565 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":291,"thread_start_us":145,"threads_started":1}
I20260810 22:32:48.907750 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.902398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:34249 (local address 127.1.221.129:43749)
0810 22:32:48.902505 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:48.902510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:48.903056 (+ 546us) server_negotiation.cc:407] Connection header received
0810 22:32:48.903164 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:48.903166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:48.903206 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:48.903258 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:48.903616 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.904169 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.905632 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.906267 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.906404 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.906453 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:48.907518 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0810 22:32:48.907610 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:32:48.907778 23418 raft_consensus.cc:2781] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:32:48.907884 23418 raft_consensus.cc:482] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:32:48.907927 23418 raft_consensus.cc:3037] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:48.908314 23053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69f4d65a4b64f26bc522c1df0ee745c" candidate_uuid: "b36d290962464d1b8135813b736c3d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39fa4a4204d245adbe2573d32194003f" is_pre_election: true
I20260810 22:32:48.908563 23053 raft_consensus.cc:2454] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b36d290962464d1b8135813b736c3d3c in term 0.
I20260810 22:32:48.909281 23418 raft_consensus.cc:504] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d290962464d1b8135813b736c3d3c" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 44495 } } peers { permanent_uuid: "4eec9ff2f2684f0abc1a7d25867c0952" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37591 } } peers { permanent_uuid: "39fa4a4204d245adbe2573d32194003f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43749 } }
I20260810 22:32:48.909583 23418 leader_election.cc:288] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [CANDIDATE]: Term 1 election: Requested vote from peers 4eec9ff2f2684f0abc1a7d25867c0952 (127.1.221.131:37591), 39fa4a4204d245adbe2573d32194003f (127.1.221.129:43749)
I20260810 22:32:48.909996 23342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69f4d65a4b64f26bc522c1df0ee745c" candidate_uuid: "b36d290962464d1b8135813b736c3d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec9ff2f2684f0abc1a7d25867c0952"
I20260810 22:32:48.910162 23342 raft_consensus.cc:3037] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:48.911343 23342 raft_consensus.cc:2454] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b36d290962464d1b8135813b736c3d3c in term 1.
I20260810 22:32:48.911610 23053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69f4d65a4b64f26bc522c1df0ee745c" candidate_uuid: "b36d290962464d1b8135813b736c3d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39fa4a4204d245adbe2573d32194003f"
I20260810 22:32:48.911675 23124 leader_election.cc:302] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eec9ff2f2684f0abc1a7d25867c0952, b36d290962464d1b8135813b736c3d3c; no voters:
I20260810 22:32:48.911744 23053 raft_consensus.cc:3037] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:48.911880 23418 raft_consensus.cc:2781] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:48.912158 23418 raft_consensus.cc:686] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [term 1 LEADER]: Becoming Leader. State: Replica: b36d290962464d1b8135813b736c3d3c, State: Running, Role: LEADER
I20260810 22:32:48.912497 23418 consensus_queue.cc:237] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d290962464d1b8135813b736c3d3c" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 44495 } } peers { permanent_uuid: "4eec9ff2f2684f0abc1a7d25867c0952" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37591 } } peers { permanent_uuid: "39fa4a4204d245adbe2573d32194003f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43749 } }
I20260810 22:32:48.913071 23053 raft_consensus.cc:2454] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b36d290962464d1b8135813b736c3d3c in term 1.
I20260810 22:32:48.916117 22893 catalog_manager.cc:5495] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c reported cstate change: term changed from 0 to 1, leader changed from <none> to b36d290962464d1b8135813b736c3d3c (127.1.221.130). New cstate: current_term: 1 leader_uuid: "b36d290962464d1b8135813b736c3d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d290962464d1b8135813b736c3d3c" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eec9ff2f2684f0abc1a7d25867c0952" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39fa4a4204d245adbe2573d32194003f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43749 } health_report { overall_health: UNKNOWN } } }
I20260810 22:32:48.930143 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.926434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.1.221.130:44495)
0810 22:32:48.926579 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:48.926583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:48.926598 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:32:48.926731 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:48.926733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:48.926788 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:48.926899 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:32:48.927156 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.927658 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.928556 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:48.928784 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:48.928891 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:48.929220 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:32:48.929701 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:32:48.929860 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:48.929957 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 22:32:48.929994 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:32:48.936118 23342 raft_consensus.cc:1261] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952 [term 1 FOLLOWER]: Refusing update from remote peer b36d290962464d1b8135813b736c3d3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:48.936124 23053 raft_consensus.cc:1261] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f [term 1 FOLLOWER]: Refusing update from remote peer b36d290962464d1b8135813b736c3d3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:48.936926 23418 consensus_queue.cc:1035] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39fa4a4204d245adbe2573d32194003f" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 43749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:32:48.937152 23423 consensus_queue.cc:1035] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eec9ff2f2684f0abc1a7d25867c0952" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:32:48.941670 23106 tablet.cc:2369] T d69f4d65a4b64f26bc522c1df0ee745c P 39fa4a4204d245adbe2573d32194003f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:32:48.945171 23430 mvcc.cc:205] Tried to move back new op lower bound from 7317099187954745344 to 7317099187868745728. Current Snapshot: MvccSnapshot[applied={T|T < 7317099187954745344}]
I20260810 22:32:48.955653 22892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:48.955853 22892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:48.958298 22892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5bdba4d231da44118d82ceb1a61e781e: Sending DeleteTablet for 3 replicas of tablet d69f4d65a4b64f26bc522c1df0ee745c
I20260810 22:32:48.959088 23177 tablet_service.cc:1514] Processing DeleteTablet for tablet d69f4d65a4b64f26bc522c1df0ee745c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:48 UTC) from {username='slave'} at 127.0.0.1:56632
I20260810 22:32:48.959241 23031 tablet_service.cc:1514] Processing DeleteTablet for tablet d69f4d65a4b64f26bc522c1df0ee745c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:48 UTC) from {username='slave'} at 127.0.0.1:57066
I20260810 22:32:48.959234 23319 tablet_service.cc:1514] Processing DeleteTablet for tablet d69f4d65a4b64f26bc522c1df0ee745c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:48 UTC) from {username='slave'} at 127.0.0.1:47250
I20260810 22:32:48.959684 23439 tablet_replica.cc:330] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c: stopping tablet replica
I20260810 22:32:48.959854 23441 tablet_replica.cc:330] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952: stopping tablet replica
I20260810 22:32:48.960047 23439 raft_consensus.cc:2229] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:32:48.960259 23441 raft_consensus.cc:2229] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:32:48.960495 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 22960
I20260810 22:32:48.960506 23439 raft_consensus.cc:2258] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:48.963534 23439 ts_tablet_manager.cc:1905] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:48.966777 23439 ts_tablet_manager.cc:1918] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:48.966863 23439 log.cc:1196] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/d69f4d65a4b64f26bc522c1df0ee745c
I20260810 22:32:48.967159 23439 ts_tablet_manager.cc:1939] T d69f4d65a4b64f26bc522c1df0ee745c P b36d290962464d1b8135813b736c3d3c: Deleting consensus metadata
I20260810 22:32:48.968535 22876 catalog_manager.cc:4841] TS b36d290962464d1b8135813b736c3d3c (127.1.221.130:44495): tablet d69f4d65a4b64f26bc522c1df0ee745c (table test-table [id=88cdfd4e931347fb82ac878aeb433899]) successfully deleted
I20260810 22:32:48.973263 23441 raft_consensus.cc:2258] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:48.975869 23441 ts_tablet_manager.cc:1905] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:48.978863 23441 ts_tablet_manager.cc:1918] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:48.978993 23441 log.cc:1196] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/d69f4d65a4b64f26bc522c1df0ee745c
I20260810 22:32:48.979295 23441 ts_tablet_manager.cc:1939] T d69f4d65a4b64f26bc522c1df0ee745c P 4eec9ff2f2684f0abc1a7d25867c0952: Deleting consensus metadata
I20260810 22:32:48.980588 22875 catalog_manager.cc:4841] TS 4eec9ff2f2684f0abc1a7d25867c0952 (127.1.221.131:37591): tablet d69f4d65a4b64f26bc522c1df0ee745c (table test-table [id=88cdfd4e931347fb82ac878aeb433899]) successfully deleted
W20260810 22:32:48.982216 22878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:43749 (error 108)
I20260810 22:32:48.982410 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 23108
I20260810 22:32:48.985112 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:48.984746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:43749 (local address 127.0.0.1:57070)
0810 22:32:48.984922 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:48.985004 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:43749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 22:32:48.985311 22878 catalog_manager.cc:4584] TS 39fa4a4204d245adbe2573d32194003f (127.1.221.129:43749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d69f4d65a4b64f26bc522c1df0ee745c: Network error: Client connection negotiation failed: client connection to 127.1.221.129:43749: connect: Connection refused (error 111)
I20260810 22:32:48.990239 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 23253
I20260810 22:32:48.997454 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 22860
2026-08-10T22:32:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1060 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:32:49 dist-test-slave-njxd krb5kdc[23445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:32:49 dist-test-slave-njxd krb5kdc[23445](info): set up 2 sockets
Aug 10 22:32:49 dist-test-slave-njxd krb5kdc[23445](info): commencing operation
krb5kdc: starting...
W20260810 22:32:50.090436 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:32:50 dist-test-slave-njxd krb5kdc[23445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:32:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:32:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:32:50.216918 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:32851
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:32851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:50.333832 23459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:50.334165 23459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:50.334240 23459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:50.338294 23459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:32:50.338397 23459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:50.338438 23459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:50.338476 23459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:32:50.338510 23459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:32:50.343938 23459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:32851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:32851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:50.345114 23459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:50.346239 23459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:50.355201 23464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:50.355166 23467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:50.355175 23465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:50.355408 23459 server_base.cc:1034] running on GCE node
I20260810 22:32:50.356086 23459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:50.357234 23459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:50.358400 23459 hybrid_clock.cc:648] HybridClock initialized: now 1786401170358381 us; error 30 us; skew 500 ppm
Aug 10 22:32:50 dist-test-slave-njxd krb5kdc[23445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401170, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:50.361272 23459 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:32:50.362234 23459 webserver.cc:460] Webserver started at http://127.1.221.190:45753/ using document root <none> and password file <none>
I20260810 22:32:50.362802 23459 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:50.362864 23459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:50.363058 23459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:50.365211 23459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "3ffe583612ad47cbb54ac3d083b5a8c3"
format_stamp: "Formatted at 2026-08-10 22:32:50 on dist-test-slave-njxd"
server_key: "29f3660cb9ecbd658d264e82d3374753"
server_key_iv: "c2845c41f0bf227844b5ae3ed2af23d1"
server_key_version: "encryptionkey@0"
I20260810 22:32:50.365813 23459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "3ffe583612ad47cbb54ac3d083b5a8c3"
format_stamp: "Formatted at 2026-08-10 22:32:50 on dist-test-slave-njxd"
server_key: "29f3660cb9ecbd658d264e82d3374753"
server_key_iv: "c2845c41f0bf227844b5ae3ed2af23d1"
server_key_version: "encryptionkey@0"
I20260810 22:32:50.369804 23459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:32:50.372283 23474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:50.373561 23459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:50.373775 23459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "3ffe583612ad47cbb54ac3d083b5a8c3"
format_stamp: "Formatted at 2026-08-10 22:32:50 on dist-test-slave-njxd"
server_key: "29f3660cb9ecbd658d264e82d3374753"
server_key_iv: "c2845c41f0bf227844b5ae3ed2af23d1"
server_key_version: "encryptionkey@0"
I20260810 22:32:50.373903 23459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:50.389521 23459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:50.392978 23459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:50.393226 23459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:50.400826 23459 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:32851
I20260810 22:32:50.400857 23530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:32851 every 8 connection(s)
I20260810 22:32:50.401882 23459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:32:50.403996 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 23459
I20260810 22:32:50.404131 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:32:50.404469 1910 external_mini_cluster.cc:1428] Setting key 03d94c2693c6974fa70c64a8f91d6d79
I20260810 22:32:50.404954 23532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:50.411162 23532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3: Bootstrap starting.
Aug 10 22:32:50 dist-test-slave-njxd krb5kdc[23445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:32:50.413805 23532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:50.414616 23532 log.cc:824] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:50.416632 23532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3: No bootstrap required, opened a new log
I20260810 22:32:50.418849 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:50.406028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44930 (local address 127.1.221.190:32851)
0810 22:32:50.406494 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:50.406510 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:50.406546 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:32:50.407204 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:50.407220 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:50.407543 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:50.407889 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:50.408721 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:50.409815 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:50.410681 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:50.411161 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:50.411329 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:50.414175 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:50.414180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:50.414194 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:50.414222 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:50.416020 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:50.416483 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:50.416489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:50.416493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:50.416613 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:50.416919 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:50.416924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:50.416926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:50.417321 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:50.417529 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:50.417850 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 22:32:50.418136 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":163,"threads_started":1}
I20260810 22:32:50.420676 23532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffe583612ad47cbb54ac3d083b5a8c3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 32851 } }
I20260810 22:32:50.420926 23532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:50.421063 23532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffe583612ad47cbb54ac3d083b5a8c3, State: Initialized, Role: FOLLOWER
I20260810 22:32:50.421535 23532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffe583612ad47cbb54ac3d083b5a8c3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 32851 } }
I20260810 22:32:50.421756 23532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:32:50.421829 23532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:32:50.421943 23532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:50.422827 23532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffe583612ad47cbb54ac3d083b5a8c3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 32851 } }
I20260810 22:32:50.423161 23532 leader_election.cc:302] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ffe583612ad47cbb54ac3d083b5a8c3; no voters:
I20260810 22:32:50.423462 23532 leader_election.cc:288] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:32:50.423635 23537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:50.423885 23537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [term 1 LEADER]: Becoming Leader. State: Replica: 3ffe583612ad47cbb54ac3d083b5a8c3, State: Running, Role: LEADER
I20260810 22:32:50.424268 23537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffe583612ad47cbb54ac3d083b5a8c3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 32851 } }
I20260810 22:32:50.424661 23532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:32:50.425854 23538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ffe583612ad47cbb54ac3d083b5a8c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffe583612ad47cbb54ac3d083b5a8c3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 32851 } } }
I20260810 22:32:50.425990 23538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:50.426396 23547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:32:50.426391 23539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ffe583612ad47cbb54ac3d083b5a8c3. Latest consensus state: current_term: 1 leader_uuid: "3ffe583612ad47cbb54ac3d083b5a8c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffe583612ad47cbb54ac3d083b5a8c3" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 32851 } } }
I20260810 22:32:50.426648 23539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:50.429297 23547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:32:50.435434 23547 catalog_manager.cc:1347] Generated new cluster ID: 930dff72f86e4d1793d4fc1251386b3d
I20260810 22:32:50.435500 23547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:32:50.448199 23547 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:32:50.449663 23547 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:32:50.454855 23547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3: Generated new TSK 0
I20260810 22:32:50.455687 23547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.221.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
Entry for principal kudu/127.1.221.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab.
I20260810 22:32:50.487016 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:32851
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:50.603797 23560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:50.604104 23560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:50.604185 23560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:50.608186 23560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:50.608282 23560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:50.608373 23560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:32:50.613682 23560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.129.keytab
--principal=kudu/127.1.221.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:32851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:50.614813 23560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:50.615885 23560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:50.625846 23565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:50.625895 23568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:50.625881 23566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:50.626214 23560 server_base.cc:1034] running on GCE node
I20260810 22:32:50.626688 23560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:50.627290 23560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:50.628453 23560 hybrid_clock.cc:648] HybridClock initialized: now 1786401170628434 us; error 31 us; skew 500 ppm
Aug 10 22:32:50 dist-test-slave-njxd krb5kdc[23445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401170, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:50.631397 23560 init.cc:377] Logged in from keytab as kudu/127.1.221.129@KRBTEST.COM (short username kudu)
I20260810 22:32:50.632227 23560 webserver.cc:460] Webserver started at http://127.1.221.129:36667/ using document root <none> and password file <none>
I20260810 22:32:50.632777 23560 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:50.632840 23560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:50.633011 23560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:50.635146 23560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "5f431ea235b14bfb85369b39bb83bfab"
format_stamp: "Formatted at 2026-08-10 22:32:50 on dist-test-slave-njxd"
server_key: "3eb36eaa46c8651e2452e578ca8791bb"
server_key_iv: "85b4198f93d5825f0e1d8ecd0c526409"
server_key_version: "encryptionkey@0"
I20260810 22:32:50.635686 23560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "5f431ea235b14bfb85369b39bb83bfab"
format_stamp: "Formatted at 2026-08-10 22:32:50 on dist-test-slave-njxd"
server_key: "3eb36eaa46c8651e2452e578ca8791bb"
server_key_iv: "85b4198f93d5825f0e1d8ecd0c526409"
server_key_version: "encryptionkey@0"
I20260810 22:32:50.639457 23560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:32:50.641966 23576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:50.643217 23560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:32:50.643368 23560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "5f431ea235b14bfb85369b39bb83bfab"
format_stamp: "Formatted at 2026-08-10 22:32:50 on dist-test-slave-njxd"
server_key: "3eb36eaa46c8651e2452e578ca8791bb"
server_key_iv: "85b4198f93d5825f0e1d8ecd0c526409"
server_key_version: "encryptionkey@0"
I20260810 22:32:50.643491 23560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:50.667035 23560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:50.670588 23560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:50.670814 23560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:50.671465 23560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:50.672345 23560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:50.672416 23560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:50.672487 23560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:50.672538 23560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:50.682415 23560 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:41719
I20260810 22:32:50.682439 23699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:41719 every 8 connection(s)
I20260810 22:32:50.683331 23560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:32:50.685003 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 23560
I20260810 22:32:50.685159 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:32:50.685484 1910 external_mini_cluster.cc:1428] Setting key 149944806ce24f340e78cf52e0adbb91
Aug 10 22:32:50 dist-test-slave-njxd krb5kdc[23445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401170, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.129@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.130@KRBTEST.COM; defaulting to no policy
I20260810 22:32:50.698793 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:50.685298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:42459 (local address 127.1.221.190:32851)
0810 22:32:50.685433 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:50.685445 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:50.686514 (+ 1069us) server_negotiation.cc:407] Connection header received
0810 22:32:50.687967 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:50.687971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:50.688022 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:50.688099 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:50.689454 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:50.690049 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:50.691820 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:50.692091 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:50.692209 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:50.695102 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:50.695105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:50.695107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:50.695131 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:50.696710 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:50.697230 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:50.697234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:50.697236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:50.697317 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:50.697685 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:50.697689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:50.697691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:50.697870 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:50.697979 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:50.698482 (+ 503us) server_negotiation.cc:299] Negotiation successful
0810 22:32:50.698609 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.130@KRBTEST.COM" created.
I20260810 22:32:50.699743 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:50.685813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:32851 (local address 127.1.221.129:42459)
0810 22:32:50.686376 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:50.686419 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:50.687753 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:50.688272 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:50.688279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:50.688630 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:50.689285 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:50.689306 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:50.690182 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:50.690185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:50.691680 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:50.691695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:50.692286 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:50.692294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:50.692536 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:50.693171 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:50.693198 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:50.694929 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:50.696833 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:50.696839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:50.696854 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:50.697113 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:50.697413 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:50.697416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:50.697418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:50.697563 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:50.697978 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:50.697984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:50.698238 (+ 254us) client_negotiation.cc:769] Sending connection context
0810 22:32:50.698423 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 22:32:50.698749 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":139,"threads_started":1}
I20260810 22:32:50.701122 23700 heartbeater.cc:347] Connected to a master server at 127.1.221.190:32851
I20260810 22:32:50.701417 23700 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:50.702102 23700 heartbeater.cc:511] Master 127.1.221.190:32851 requested a full tablet report, sending...
I20260810 22:32:50.703920 23492 ts_manager.cc:194] Registered new tserver with Master: 5f431ea235b14bfb85369b39bb83bfab (127.1.221.129:41719)
I20260810 22:32:50.705739 23492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.129@KRBTEST.COM'} at 127.1.221.129:42459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
Entry for principal kudu/127.1.221.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab.
I20260810 22:32:50.717252 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:32851
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:50.833192 23705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:50.833436 23705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:50.833475 23705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:50.837414 23705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:50.837471 23705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:50.837558 23705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:32:50.842780 23705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.130.keytab
--principal=kudu/127.1.221.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:32851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:50.843881 23705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:50.845216 23705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:50.855726 23711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:50.855734 23714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:50.855746 23712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:50.856036 23705 server_base.cc:1034] running on GCE node
I20260810 22:32:50.856460 23705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:50.857110 23705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:50.858300 23705 hybrid_clock.cc:648] HybridClock initialized: now 1786401170858289 us; error 32 us; skew 500 ppm
Aug 10 22:32:50 dist-test-slave-njxd krb5kdc[23445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401170, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:50.861043 23705 init.cc:377] Logged in from keytab as kudu/127.1.221.130@KRBTEST.COM (short username kudu)
I20260810 22:32:50.861934 23705 webserver.cc:460] Webserver started at http://127.1.221.130:36877/ using document root <none> and password file <none>
I20260810 22:32:50.862469 23705 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:50.862528 23705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:50.862694 23705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:50.864724 23705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "0170d6620fa4468c9998a705977ce296"
format_stamp: "Formatted at 2026-08-10 22:32:50 on dist-test-slave-njxd"
server_key: "2a896a0c961f95e1b63198484a574369"
server_key_iv: "f3ea10db5c835efb53a55dccd69d108e"
server_key_version: "encryptionkey@0"
I20260810 22:32:50.865257 23705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "0170d6620fa4468c9998a705977ce296"
format_stamp: "Formatted at 2026-08-10 22:32:50 on dist-test-slave-njxd"
server_key: "2a896a0c961f95e1b63198484a574369"
server_key_iv: "f3ea10db5c835efb53a55dccd69d108e"
server_key_version: "encryptionkey@0"
I20260810 22:32:50.868969 23705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 22:32:50.871382 23721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:50.872677 23705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:32:50.872803 23705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "0170d6620fa4468c9998a705977ce296"
format_stamp: "Formatted at 2026-08-10 22:32:50 on dist-test-slave-njxd"
server_key: "2a896a0c961f95e1b63198484a574369"
server_key_iv: "f3ea10db5c835efb53a55dccd69d108e"
server_key_version: "encryptionkey@0"
I20260810 22:32:50.872912 23705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:50.888113 23705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:50.891059 23705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:50.891275 23705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:50.891892 23705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:50.892733 23705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:50.892781 23705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:50.892853 23705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:50.892891 23705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:50.901803 23705 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:42785
I20260810 22:32:50.901892 23842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:42785 every 8 connection(s)
I20260810 22:32:50.902772 23705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:32:50.904774 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 23705
I20260810 22:32:50.904927 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:32:50.905226 1910 external_mini_cluster.cc:1428] Setting key 00a34026bc35bfcb9c1bb262607d6943
Aug 10 22:32:50 dist-test-slave-njxd krb5kdc[23445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401170, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.130@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.221.131@KRBTEST.COM; defaulting to no policy
I20260810 22:32:50.918622 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:50.904542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:51039 (local address 127.1.221.190:32851)
0810 22:32:50.904740 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:50.904745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:50.905732 (+ 987us) server_negotiation.cc:407] Connection header received
0810 22:32:50.907417 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:50.907419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:50.907463 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:50.907534 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:50.909236 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:50.910081 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:50.911199 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:50.911458 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:50.911548 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:50.914514 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:50.914519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:50.914527 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:50.914593 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:50.916438 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:50.916948 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:50.916953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:50.916956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:50.917056 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:50.917408 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:50.917415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:50.917418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:50.917691 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:50.917785 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:50.918331 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 22:32:50.918456 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 22:32:50.919646 23845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:50.904981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:32851 (local address 127.1.221.130:51039)
0810 22:32:50.905534 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:50.905631 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:50.907173 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:50.907761 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:50.907773 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:50.908319 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:50.909039 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:50.909063 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:50.910204 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:50.910207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:50.911082 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:50.911092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:50.911628 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:50.911634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:50.911880 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:50.912511 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:50.912536 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:50.914344 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:50.916567 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:50.916572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:50.916585 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:50.916842 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:50.917168 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:50.917170 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:50.917171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:50.917306 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:50.917799 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:50.917803 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:50.918086 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 22:32:50.918286 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 22:32:50.918602 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.131@KRBTEST.COM" created.
I20260810 22:32:50.920979 23843 heartbeater.cc:347] Connected to a master server at 127.1.221.190:32851
I20260810 22:32:50.921223 23843 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:50.921782 23843 heartbeater.cc:511] Master 127.1.221.190:32851 requested a full tablet report, sending...
I20260810 22:32:50.922930 23492 ts_manager.cc:194] Registered new tserver with Master: 0170d6620fa4468c9998a705977ce296 (127.1.221.130:42785)
I20260810 22:32:50.923516 23492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.130@KRBTEST.COM'} at 127.1.221.130:51039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
Entry for principal kudu/127.1.221.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab.
I20260810 22:32:50.939129 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:32851
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:51.069298 23848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:51.069741 23848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:51.069808 23848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:51.076776 23848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:51.076946 23848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:51.077088 23848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:32:51.082684 23848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.131.keytab
--principal=kudu/127.1.221.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:32851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:51.083695 23848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:51.084718 23848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:51.094321 23856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:51.094322 23853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:51.094319 23854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:51.094515 23848 server_base.cc:1034] running on GCE node
I20260810 22:32:51.094982 23848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:51.095588 23848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:51.096767 23848 hybrid_clock.cc:648] HybridClock initialized: now 1786401171096760 us; error 32 us; skew 500 ppm
Aug 10 22:32:51 dist-test-slave-njxd krb5kdc[23445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401171, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:51.099678 23848 init.cc:377] Logged in from keytab as kudu/127.1.221.131@KRBTEST.COM (short username kudu)
I20260810 22:32:51.100481 23848 webserver.cc:460] Webserver started at http://127.1.221.131:33641/ using document root <none> and password file <none>
I20260810 22:32:51.100993 23848 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:51.101049 23848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:51.101212 23848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:51.103365 23848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "587bd4a07cd64f8783fd849d8f21a856"
format_stamp: "Formatted at 2026-08-10 22:32:51 on dist-test-slave-njxd"
server_key: "a9c825afb087d36c74d25b855324659e"
server_key_iv: "98aa22a8aa5c8863bf37f0102cc8633f"
server_key_version: "encryptionkey@0"
I20260810 22:32:51.103982 23848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "587bd4a07cd64f8783fd849d8f21a856"
format_stamp: "Formatted at 2026-08-10 22:32:51 on dist-test-slave-njxd"
server_key: "a9c825afb087d36c74d25b855324659e"
server_key_iv: "98aa22a8aa5c8863bf37f0102cc8633f"
server_key_version: "encryptionkey@0"
I20260810 22:32:51.107649 23848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 22:32:51.109906 23863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:51.111168 23848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:32:51.111409 23848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "587bd4a07cd64f8783fd849d8f21a856"
format_stamp: "Formatted at 2026-08-10 22:32:51 on dist-test-slave-njxd"
server_key: "a9c825afb087d36c74d25b855324659e"
server_key_iv: "98aa22a8aa5c8863bf37f0102cc8633f"
server_key_version: "encryptionkey@0"
I20260810 22:32:51.111517 23848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:51.126024 23848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:51.128895 23848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:51.129045 23848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:51.129637 23848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:51.130445 23848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:51.130491 23848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:51.130529 23848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:51.130548 23848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:51.139320 23848 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:34819
I20260810 22:32:51.139372 23986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:34819 every 8 connection(s)
I20260810 22:32:51.140214 23848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:32:51.146595 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 23848
I20260810 22:32:51.146740 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:32:51.147092 1910 external_mini_cluster.cc:1428] Setting key 83e20f859aadf9465ef871af790e4fb4
Aug 10 22:32:51 dist-test-slave-njxd krb5kdc[23445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401171, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.131@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:32:51.154366 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.142091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:60051 (local address 127.1.221.190:32851)
0810 22:32:51.142234 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:51.142238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:51.143043 (+ 805us) server_negotiation.cc:407] Connection header received
0810 22:32:51.144328 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:51.144332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:51.144390 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:51.144473 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:51.145643 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.146166 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.147554 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.147874 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.148010 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.150782 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:51.150785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:51.150788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:51.150816 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:51.152135 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:51.152680 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:51.152684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:51.152686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:51.152777 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:51.153129 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:51.153133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:51.153135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:51.153326 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:51.153436 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:51.154067 (+ 631us) server_negotiation.cc:299] Negotiation successful
0810 22:32:51.154193 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:32:51.155189 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.142454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:32851 (local address 127.1.221.131:60051)
0810 22:32:51.142901 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:51.142943 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:51.144121 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:51.144654 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:51.144661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:51.145030 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:51.145483 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.145496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.146305 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.146309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.147380 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.147391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.148080 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.148084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.148263 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.148963 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:32:51.148984 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:51.150623 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:51.152279 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:51.152286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:51.152302 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:51.152567 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:51.152883 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:32:51.152887 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:32:51.152889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:32:51.153030 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:32:51.153436 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:51.153444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:51.153803 (+ 359us) client_negotiation.cc:769] Sending connection context
0810 22:32:51.154011 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 22:32:51.154333 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260810 22:32:51.156261 23987 heartbeater.cc:347] Connected to a master server at 127.1.221.190:32851
I20260810 22:32:51.156525 23987 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:51.157032 23987 heartbeater.cc:511] Master 127.1.221.190:32851 requested a full tablet report, sending...
I20260810 22:32:51.158087 23492 ts_manager.cc:194] Registered new tserver with Master: 587bd4a07cd64f8783fd849d8f21a856 (127.1.221.131:34819)
I20260810 22:32:51.158658 23492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.221.131@KRBTEST.COM'} at 127.1.221.131:60051
I20260810 22:32:51.161131 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:32:51.170665 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.163227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.1.221.190:32851)
0810 22:32:51.163479 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:51.163485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:51.163661 (+ 176us) server_negotiation.cc:407] Connection header received
0810 22:32:51.163832 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:51.163835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:51.163881 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:51.163965 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:51.164689 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.165189 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.166109 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.166335 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.166438 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.167749 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:51.167760 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:51.167762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:51.167785 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:51.168959 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:51.169370 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:51.169374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:51.169375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:51.169455 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:51.169766 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:51.169769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:51.169771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:51.169981 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:51.170127 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:51.170369 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 22:32:51.170499 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:32:51.173382 23492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:32:51.176548 23492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:32:51.191015 24003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.185448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.1.221.129:41719)
0810 22:32:51.185781 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:51.185798 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:51.185996 (+ 198us) server_negotiation.cc:407] Connection header received
0810 22:32:51.186548 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:51.186555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:51.186740 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:51.186935 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:51.187334 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.188114 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.189641 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.190311 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.190425 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.190598 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:51.190757 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 22:32:51.190860 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":141,"threads_started":1}
I20260810 22:32:51.190997 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.184804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41719 (local address 127.0.0.1:37394)
0810 22:32:51.185915 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:51.185929 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:51.186054 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:51.187090 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:51.187095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:51.187116 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:51.187222 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.187226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.188277 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.188280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.189476 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.189487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.190429 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.190431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.190536 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.190586 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 22:32:51.190719 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 22:32:51.190849 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":70,"thread_start_us":93,"threads_started":1}
I20260810 22:32:51.191890 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.184784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.1.221.131:34819)
0810 22:32:51.185168 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:51.185182 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:51.185906 (+ 724us) server_negotiation.cc:407] Connection header received
0810 22:32:51.186213 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:51.186220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:51.186446 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:51.186603 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:51.186971 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.188206 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.190178 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.191038 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.191144 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.191281 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:51.191621 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 22:32:51.191742 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":166,"threads_started":1}
I20260810 22:32:51.192449 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.185376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:42785 (local address 127.0.0.1:53764)
0810 22:32:51.186266 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:51.186280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:51.186365 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:51.186908 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:51.186912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:51.186930 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:51.187035 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.187042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.188374 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.188378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.190034 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.190046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.192076 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.192079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.192181 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.192210 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 22:32:51.192289 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 22:32:51.192340 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":137,"thread_start_us":108,"threads_started":1}
I20260810 22:32:51.192579 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.185857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.1.221.130:42785)
0810 22:32:51.186330 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:51.186339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:51.186357 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:32:51.186481 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:51.186485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:51.186719 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:51.186901 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:51.187135 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.188217 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.190675 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.191704 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.191804 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.191941 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:51.192328 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 22:32:51.192429 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":105,"threads_started":1}
I20260810 22:32:51.192448 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.184604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:34819 (local address 127.0.0.1:48506)
0810 22:32:51.185760 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:51.185816 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:51.186023 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:51.186650 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:51.186657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:51.186678 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:51.186826 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.186835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.188381 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.188384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.190034 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.190046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.191352 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.191356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.191451 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.191474 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:32:51.192217 (+ 743us) client_negotiation.cc:240] Negotiation successful
0810 22:32:51.192311 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":246,"thread_start_us":91,"threads_started":1}
I20260810 22:32:51.194449 23629 tablet_service.cc:1467] Processing CreateTablet for tablet 451f87866422484292bc3b503edadf30 (DEFAULT_TABLE table=test-table [id=820374a6bf4a4c8e8ddab282b938668a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:51.195698 23629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451f87866422484292bc3b503edadf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:51.195254 23772 tablet_service.cc:1467] Processing CreateTablet for tablet 451f87866422484292bc3b503edadf30 (DEFAULT_TABLE table=test-table [id=820374a6bf4a4c8e8ddab282b938668a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:51.196274 23772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451f87866422484292bc3b503edadf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:51.196002 23915 tablet_service.cc:1467] Processing CreateTablet for tablet 451f87866422484292bc3b503edadf30 (DEFAULT_TABLE table=test-table [id=820374a6bf4a4c8e8ddab282b938668a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:51.196946 23915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451f87866422484292bc3b503edadf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:51.202646 24005 tablet_bootstrap.cc:492] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296: Bootstrap starting.
I20260810 22:32:51.204214 24006 tablet_bootstrap.cc:492] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab: Bootstrap starting.
I20260810 22:32:51.205308 24005 tablet_bootstrap.cc:654] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:51.206117 24006 tablet_bootstrap.cc:654] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:51.206336 24005 log.cc:824] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:51.206998 24006 log.cc:824] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:51.207057 24007 tablet_bootstrap.cc:492] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856: Bootstrap starting.
I20260810 22:32:51.208709 24007 tablet_bootstrap.cc:654] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:51.208824 24006 tablet_bootstrap.cc:492] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab: No bootstrap required, opened a new log
I20260810 22:32:51.209015 24006 ts_tablet_manager.cc:1397] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 22:32:51.209856 24007 log.cc:824] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:51.211447 24005 tablet_bootstrap.cc:492] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296: No bootstrap required, opened a new log
I20260810 22:32:51.211647 24005 ts_tablet_manager.cc:1397] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260810 22:32:51.211860 24007 tablet_bootstrap.cc:492] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856: No bootstrap required, opened a new log
I20260810 22:32:51.212064 24007 ts_tablet_manager.cc:1397] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 22:32:51.214738 24006 raft_consensus.cc:348] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587bd4a07cd64f8783fd849d8f21a856" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34819 } } peers { permanent_uuid: "5f431ea235b14bfb85369b39bb83bfab" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41719 } } peers { permanent_uuid: "0170d6620fa4468c9998a705977ce296" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 42785 } }
I20260810 22:32:51.215167 24006 raft_consensus.cc:374] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:51.215404 24006 raft_consensus.cc:729] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f431ea235b14bfb85369b39bb83bfab, State: Initialized, Role: FOLLOWER
I20260810 22:32:51.215649 24005 raft_consensus.cc:348] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587bd4a07cd64f8783fd849d8f21a856" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34819 } } peers { permanent_uuid: "5f431ea235b14bfb85369b39bb83bfab" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41719 } } peers { permanent_uuid: "0170d6620fa4468c9998a705977ce296" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 42785 } }
I20260810 22:32:51.215965 24005 raft_consensus.cc:374] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:51.216054 24005 raft_consensus.cc:729] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0170d6620fa4468c9998a705977ce296, State: Initialized, Role: FOLLOWER
I20260810 22:32:51.215961 24007 raft_consensus.cc:348] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587bd4a07cd64f8783fd849d8f21a856" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34819 } } peers { permanent_uuid: "5f431ea235b14bfb85369b39bb83bfab" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41719 } } peers { permanent_uuid: "0170d6620fa4468c9998a705977ce296" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 42785 } }
I20260810 22:32:51.216398 24007 raft_consensus.cc:374] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:51.216470 24007 raft_consensus.cc:729] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587bd4a07cd64f8783fd849d8f21a856, State: Initialized, Role: FOLLOWER
I20260810 22:32:51.216203 24006 consensus_queue.cc:260] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587bd4a07cd64f8783fd849d8f21a856" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34819 } } peers { permanent_uuid: "5f431ea235b14bfb85369b39bb83bfab" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41719 } } peers { permanent_uuid: "0170d6620fa4468c9998a705977ce296" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 42785 } }
I20260810 22:32:51.216565 24005 consensus_queue.cc:260] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587bd4a07cd64f8783fd849d8f21a856" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34819 } } peers { permanent_uuid: "5f431ea235b14bfb85369b39bb83bfab" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41719 } } peers { permanent_uuid: "0170d6620fa4468c9998a705977ce296" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 42785 } }
I20260810 22:32:51.216928 24007 consensus_queue.cc:260] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587bd4a07cd64f8783fd849d8f21a856" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34819 } } peers { permanent_uuid: "5f431ea235b14bfb85369b39bb83bfab" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41719 } } peers { permanent_uuid: "0170d6620fa4468c9998a705977ce296" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 42785 } }
I20260810 22:32:51.217650 24005 ts_tablet_manager.cc:1428] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:32:51.217664 24006 ts_tablet_manager.cc:1428] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260810 22:32:51.217844 24007 ts_tablet_manager.cc:1428] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:32:51.218096 23843 heartbeater.cc:503] Master 127.1.221.190:32851 was elected leader, sending a full tablet report...
I20260810 22:32:51.218194 23987 heartbeater.cc:503] Master 127.1.221.190:32851 was elected leader, sending a full tablet report...
I20260810 22:32:51.219069 23700 heartbeater.cc:503] Master 127.1.221.190:32851 was elected leader, sending a full tablet report...
I20260810 22:32:51.355834 24011 raft_consensus.cc:482] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:32:51.356140 24011 raft_consensus.cc:504] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587bd4a07cd64f8783fd849d8f21a856" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34819 } } peers { permanent_uuid: "5f431ea235b14bfb85369b39bb83bfab" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41719 } } peers { permanent_uuid: "0170d6620fa4468c9998a705977ce296" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 42785 } }
I20260810 22:32:51.357337 24011 leader_election.cc:288] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 587bd4a07cd64f8783fd849d8f21a856 (127.1.221.131:34819), 0170d6620fa4468c9998a705977ce296 (127.1.221.130:42785)
I20260810 22:32:51.361457 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.357769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:34137 (local address 127.1.221.131:34819)
0810 22:32:51.357929 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:51.357936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:51.358108 (+ 172us) server_negotiation.cc:407] Connection header received
0810 22:32:51.358247 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:51.358250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:51.358310 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:51.358385 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:51.358722 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.359256 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.360472 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.360881 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.360999 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.361081 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:51.361244 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 22:32:51.361331 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 22:32:51.361469 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.357637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:34819 (local address 127.1.221.129:34137)
0810 22:32:51.357982 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:51.357999 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:51.358137 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:51.358418 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:51.358422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:51.358443 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:51.358592 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.358599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.359379 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.359384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.360341 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.360350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.361003 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.361010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.361104 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.361134 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 22:32:51.361214 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 22:32:51.361270 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":58,"thread_start_us":135,"threads_started":1}
I20260810 22:32:51.361611 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.357782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:58525 (local address 127.1.221.130:42785)
0810 22:32:51.357995 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:51.358000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:51.358245 (+ 245us) server_negotiation.cc:407] Connection header received
0810 22:32:51.358333 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:51.358335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:51.358406 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:51.358489 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:51.358781 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.359261 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.360505 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.360902 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.361058 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.361112 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:51.361379 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 22:32:51.361454 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260810 22:32:51.361832 24015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.357662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:42785 (local address 127.1.221.129:58525)
0810 22:32:51.358120 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:51.358135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:51.358248 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:51.358512 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:51.358515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:51.358531 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:51.358644 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.358669 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.359383 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.359388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.360347 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.360356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.361092 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.361099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.361238 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.361263 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 22:32:51.361339 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 22:32:51.361403 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":50,"thread_start_us":107,"threads_started":1}
I20260810 22:32:51.362353 23937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451f87866422484292bc3b503edadf30" candidate_uuid: "5f431ea235b14bfb85369b39bb83bfab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "587bd4a07cd64f8783fd849d8f21a856" is_pre_election: true
I20260810 22:32:51.362479 23794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451f87866422484292bc3b503edadf30" candidate_uuid: "5f431ea235b14bfb85369b39bb83bfab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0170d6620fa4468c9998a705977ce296" is_pre_election: true
I20260810 22:32:51.362699 23937 raft_consensus.cc:2454] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f431ea235b14bfb85369b39bb83bfab in term 0.
I20260810 22:32:51.362856 23794 raft_consensus.cc:2454] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f431ea235b14bfb85369b39bb83bfab in term 0.
I20260810 22:32:51.363214 23577 leader_election.cc:302] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 587bd4a07cd64f8783fd849d8f21a856, 5f431ea235b14bfb85369b39bb83bfab; no voters:
I20260810 22:32:51.363502 24011 raft_consensus.cc:2781] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:32:51.363590 24011 raft_consensus.cc:482] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:32:51.363641 24011 raft_consensus.cc:3037] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:51.364704 24011 raft_consensus.cc:504] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587bd4a07cd64f8783fd849d8f21a856" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34819 } } peers { permanent_uuid: "5f431ea235b14bfb85369b39bb83bfab" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41719 } } peers { permanent_uuid: "0170d6620fa4468c9998a705977ce296" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 42785 } }
I20260810 22:32:51.365054 24011 leader_election.cc:288] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [CANDIDATE]: Term 1 election: Requested vote from peers 587bd4a07cd64f8783fd849d8f21a856 (127.1.221.131:34819), 0170d6620fa4468c9998a705977ce296 (127.1.221.130:42785)
I20260810 22:32:51.365406 23937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451f87866422484292bc3b503edadf30" candidate_uuid: "5f431ea235b14bfb85369b39bb83bfab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "587bd4a07cd64f8783fd849d8f21a856"
I20260810 22:32:51.365458 23794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "451f87866422484292bc3b503edadf30" candidate_uuid: "5f431ea235b14bfb85369b39bb83bfab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0170d6620fa4468c9998a705977ce296"
I20260810 22:32:51.365559 23937 raft_consensus.cc:3037] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:51.365624 23794 raft_consensus.cc:3037] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:51.366685 23937 raft_consensus.cc:2454] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f431ea235b14bfb85369b39bb83bfab in term 1.
I20260810 22:32:51.366760 23794 raft_consensus.cc:2454] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f431ea235b14bfb85369b39bb83bfab in term 1.
I20260810 22:32:51.367010 23577 leader_election.cc:302] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 587bd4a07cd64f8783fd849d8f21a856, 5f431ea235b14bfb85369b39bb83bfab; no voters:
I20260810 22:32:51.367221 24011 raft_consensus.cc:2781] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:51.367455 24011 raft_consensus.cc:686] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [term 1 LEADER]: Becoming Leader. State: Replica: 5f431ea235b14bfb85369b39bb83bfab, State: Running, Role: LEADER
I20260810 22:32:51.367826 24011 consensus_queue.cc:237] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587bd4a07cd64f8783fd849d8f21a856" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34819 } } peers { permanent_uuid: "5f431ea235b14bfb85369b39bb83bfab" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41719 } } peers { permanent_uuid: "0170d6620fa4468c9998a705977ce296" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 42785 } }
I20260810 22:32:51.371054 23490 catalog_manager.cc:5495] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f431ea235b14bfb85369b39bb83bfab (127.1.221.129). New cstate: current_term: 1 leader_uuid: "5f431ea235b14bfb85369b39bb83bfab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587bd4a07cd64f8783fd849d8f21a856" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f431ea235b14bfb85369b39bb83bfab" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 41719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0170d6620fa4468c9998a705977ce296" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 42785 } health_report { overall_health: UNKNOWN } } }
W20260810 22:32:51.391260 23988 tablet.cc:2369] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:32:51.403970 23844 tablet.cc:2369] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:32:51.426403 24003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.422624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.1.221.129:41719)
0810 22:32:51.422758 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:51.422762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:51.422775 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:32:51.422852 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:51.422854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:51.422910 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:51.422999 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:32:51.423212 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.423856 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.424670 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.425000 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.425092 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.425412 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:32:51.425947 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:32:51.426128 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:51.426193 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 22:32:51.426255 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:32:51.432401 23937 raft_consensus.cc:1261] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856 [term 1 FOLLOWER]: Refusing update from remote peer 5f431ea235b14bfb85369b39bb83bfab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:51.432401 23794 raft_consensus.cc:1261] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296 [term 1 FOLLOWER]: Refusing update from remote peer 5f431ea235b14bfb85369b39bb83bfab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:51.433176 24016 consensus_queue.cc:1035] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [LEADER]: Connected to new peer: Peer: permanent_uuid: "587bd4a07cd64f8783fd849d8f21a856" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 34819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:32:51.433372 24011 consensus_queue.cc:1035] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0170d6620fa4468c9998a705977ce296" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 42785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:32:51.434844 23701 tablet.cc:2369] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:32:51.442127 24021 mvcc.cc:205] Tried to move back new op lower bound from 7317099198179868672 to 7317099197925208064. Current Snapshot: MvccSnapshot[applied={T|T < 7317099198179868672}]
I20260810 22:32:51.449460 23491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:51.449703 23491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:51.452195 23491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ffe583612ad47cbb54ac3d083b5a8c3: Sending DeleteTablet for 3 replicas of tablet 451f87866422484292bc3b503edadf30
I20260810 22:32:51.452912 23772 tablet_service.cc:1514] Processing DeleteTablet for tablet 451f87866422484292bc3b503edadf30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:51 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:53764
I20260810 22:32:51.452911 23629 tablet_service.cc:1514] Processing DeleteTablet for tablet 451f87866422484292bc3b503edadf30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:51 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:37394
I20260810 22:32:51.452970 23915 tablet_service.cc:1514] Processing DeleteTablet for tablet 451f87866422484292bc3b503edadf30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:51 UTC) from {username='kudu', principal='kudu/127.1.221.190@KRBTEST.COM'} at 127.0.0.1:48506
I20260810 22:32:51.453549 24029 tablet_replica.cc:330] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296: stopping tablet replica
I20260810 22:32:51.453553 24030 tablet_replica.cc:330] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856: stopping tablet replica
I20260810 22:32:51.453886 24029 raft_consensus.cc:2229] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:32:51.453890 24030 raft_consensus.cc:2229] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:32:51.454206 24029 raft_consensus.cc:2258] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:51.454317 24030 raft_consensus.cc:2258] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:51.454430 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 23560
I20260810 22:32:51.454589 24028 tablet_replica.cc:330] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab: stopping tablet replica
I20260810 22:32:51.454850 24028 raft_consensus.cc:2229] T 451f87866422484292bc3b503edadf30 P 5f431ea235b14bfb85369b39bb83bfab [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:32:51.457100 24030 ts_tablet_manager.cc:1905] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:51.459161 24029 ts_tablet_manager.cc:1905] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:51.460160 24030 ts_tablet_manager.cc:1918] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:51.460276 24030 log.cc:1196] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/451f87866422484292bc3b503edadf30
I20260810 22:32:51.460626 24030 ts_tablet_manager.cc:1939] T 451f87866422484292bc3b503edadf30 P 587bd4a07cd64f8783fd849d8f21a856: Deleting consensus metadata
I20260810 22:32:51.461944 24029 ts_tablet_manager.cc:1918] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:51.461930 23475 catalog_manager.cc:4841] TS 587bd4a07cd64f8783fd849d8f21a856 (127.1.221.131:34819): tablet 451f87866422484292bc3b503edadf30 (table test-table [id=820374a6bf4a4c8e8ddab282b938668a]) successfully deleted
I20260810 22:32:51.462045 24029 log.cc:1196] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/451f87866422484292bc3b503edadf30
I20260810 22:32:51.462339 24029 ts_tablet_manager.cc:1939] T 451f87866422484292bc3b503edadf30 P 0170d6620fa4468c9998a705977ce296: Deleting consensus metadata
I20260810 22:32:51.463382 23477 catalog_manager.cc:4841] TS 0170d6620fa4468c9998a705977ce296 (127.1.221.130:42785): tablet 451f87866422484292bc3b503edadf30 (table test-table [id=820374a6bf4a4c8e8ddab282b938668a]) successfully deleted
W20260810 22:32:51.464872 23478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:41719 (error 108)
I20260810 22:32:51.465063 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 23705
I20260810 22:32:51.466840 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.466563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:41719 (local address 127.0.0.1:37774)
0810 22:32:51.466700 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:51.466769 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 22:32:51.467026 23478 catalog_manager.cc:4584] TS 5f431ea235b14bfb85369b39bb83bfab (127.1.221.129:41719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 451f87866422484292bc3b503edadf30: Network error: Client connection negotiation failed: client connection to 127.1.221.129:41719: connect: Connection refused (error 111)
I20260810 22:32:51.472252 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 23848
I20260810 22:32:51.479876 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 23459
2026-08-10T22:32:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2484 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T22:32:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:32:51Z Disabled control of system clock
I20260810 22:32:51.513984 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:45129
--webserver_interface=127.1.221.190
--webserver_port=0
--builtin_ntp_servers=127.1.221.148:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:45129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 22:32:51.625255 24035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:51.625504 24035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:51.625540 24035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:51.629669 24035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:32:51.629737 24035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:51.629753 24035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:51.629770 24035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:32:51.629786 24035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:32:51.635042 24035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:45129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:45129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:51.636125 24035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:51.637494 24035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:32:51.646098 24035 server_base.cc:1034] running on GCE node
W20260810 22:32:51.646023 24040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:51.646042 24042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:51.646023 24044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:51.646957 24035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:51.648088 24035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:51.649266 24035 hybrid_clock.cc:648] HybridClock initialized: now 1786401171649237 us; error 39 us; skew 500 ppm
I20260810 22:32:51.651227 24035 webserver.cc:460] Webserver started at http://127.1.221.190:34867/ using document root <none> and password file <none>
I20260810 22:32:51.651824 24035 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:51.651886 24035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:51.652139 24035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:51.654300 24035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "b6440d4722354dcd901499a7d73587b9"
format_stamp: "Formatted at 2026-08-10 22:32:51 on dist-test-slave-njxd"
server_key: "b3dd65d3c12484df4c5ac70b9912fa5b"
server_key_iv: "3a79d393bfbcfd64336581b48fdc5d38"
server_key_version: "encryptionkey@0"
I20260810 22:32:51.654915 24035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "b6440d4722354dcd901499a7d73587b9"
format_stamp: "Formatted at 2026-08-10 22:32:51 on dist-test-slave-njxd"
server_key: "b3dd65d3c12484df4c5ac70b9912fa5b"
server_key_iv: "3a79d393bfbcfd64336581b48fdc5d38"
server_key_version: "encryptionkey@0"
I20260810 22:32:51.658684 24035 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:32:51.661171 24051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:51.662562 24035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:32:51.662698 24035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "b6440d4722354dcd901499a7d73587b9"
format_stamp: "Formatted at 2026-08-10 22:32:51 on dist-test-slave-njxd"
server_key: "b3dd65d3c12484df4c5ac70b9912fa5b"
server_key_iv: "3a79d393bfbcfd64336581b48fdc5d38"
server_key_version: "encryptionkey@0"
I20260810 22:32:51.662817 24035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:51.696815 24035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:51.697671 24035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:51.697885 24035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:51.705305 24035 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:45129
I20260810 22:32:51.705376 24104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:45129 every 8 connection(s)
I20260810 22:32:51.706307 24035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:32:51.709605 24105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:51.710305 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 24035
I20260810 22:32:51.710453 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:32:51.710753 1910 external_mini_cluster.cc:1428] Setting key 99f74ff9eb0eaef56670ed21b338d071
I20260810 22:32:51.716931 24105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9: Bootstrap starting.
I20260810 22:32:51.719868 24105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:51.720834 24105 log.cc:824] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:51.722126 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.712267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54560 (local address 127.1.221.190:45129)
0810 22:32:51.712972 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:51.712993 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:51.713041 (+ 48us) server_negotiation.cc:407] Connection header received
0810 22:32:51.713988 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:51.714013 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:51.714598 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:51.715025 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:51.715503 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.717106 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.718087 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.718872 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.719037 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.719869 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:51.719898 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:51.719918 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:51.720477 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:51.720547 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:51.720593 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:51.720703 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:51.720840 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 22:32:51.721125 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":98,"threads_started":1}
I20260810 22:32:51.723230 24105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9: No bootstrap required, opened a new log
I20260810 22:32:51.727262 24105 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6440d4722354dcd901499a7d73587b9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45129 } }
I20260810 22:32:51.727516 24105 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:51.727599 24105 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6440d4722354dcd901499a7d73587b9, State: Initialized, Role: FOLLOWER
I20260810 22:32:51.728045 24105 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6440d4722354dcd901499a7d73587b9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45129 } }
I20260810 22:32:51.728216 24105 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:32:51.728305 24105 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:32:51.728446 24105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:51.729436 24105 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6440d4722354dcd901499a7d73587b9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45129 } }
I20260810 22:32:51.729811 24105 leader_election.cc:302] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6440d4722354dcd901499a7d73587b9; no voters:
I20260810 22:32:51.730141 24105 leader_election.cc:288] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:32:51.730273 24110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:51.730532 24110 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [term 1 LEADER]: Becoming Leader. State: Replica: b6440d4722354dcd901499a7d73587b9, State: Running, Role: LEADER
I20260810 22:32:51.730850 24110 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6440d4722354dcd901499a7d73587b9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45129 } }
I20260810 22:32:51.731397 24105 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:32:51.732653 24111 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6440d4722354dcd901499a7d73587b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6440d4722354dcd901499a7d73587b9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45129 } } }
I20260810 22:32:51.732659 24112 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6440d4722354dcd901499a7d73587b9. Latest consensus state: current_term: 1 leader_uuid: "b6440d4722354dcd901499a7d73587b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6440d4722354dcd901499a7d73587b9" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 45129 } } }
I20260810 22:32:51.732801 24111 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:51.732815 24112 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:51.733192 24120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:32:51.736472 24120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:32:51.742393 24120 catalog_manager.cc:1347] Generated new cluster ID: a43f135a54444b31811e1fb8485b8521
I20260810 22:32:51.742470 24120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:32:51.763962 24120 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:32:51.764976 24120 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:32:51.770955 24120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9: Generated new TSK 0
I20260810 22:32:51.771734 24120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:32:51.776511 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.129:0
--local_ip_for_outbound_sockets=127.1.221.129
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45129
--builtin_ntp_servers=127.1.221.148:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 22:32:51.891724 24130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:51.891984 24130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:51.892023 24130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:51.896207 24130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:51.896265 24130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:51.896334 24130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.129
I20260810 22:32:51.901126 24130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.221.129
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.129
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:51.902285 24130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:51.903769 24130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:51.913395 24136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:51.913434 24138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:51.913395 24135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:51.913542 24130 server_base.cc:1034] running on GCE node
I20260810 22:32:51.914055 24130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:51.914687 24130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:51.915881 24130 hybrid_clock.cc:648] HybridClock initialized: now 1786401171915849 us; error 45 us; skew 500 ppm
I20260810 22:32:51.917774 24130 webserver.cc:460] Webserver started at http://127.1.221.129:37501/ using document root <none> and password file <none>
I20260810 22:32:51.918344 24130 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:51.918437 24130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:51.918678 24130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:51.920696 24130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/data/instance:
uuid: "5ac3344dbc9f46faa4995a94fe3d40a0"
format_stamp: "Formatted at 2026-08-10 22:32:51 on dist-test-slave-njxd"
server_key: "6e19da0daa40f1c37dee61bf8d5721ee"
server_key_iv: "c87b0de90806a62eaea3824de08076c3"
server_key_version: "encryptionkey@0"
I20260810 22:32:51.921268 24130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance:
uuid: "5ac3344dbc9f46faa4995a94fe3d40a0"
format_stamp: "Formatted at 2026-08-10 22:32:51 on dist-test-slave-njxd"
server_key: "6e19da0daa40f1c37dee61bf8d5721ee"
server_key_iv: "c87b0de90806a62eaea3824de08076c3"
server_key_version: "encryptionkey@0"
I20260810 22:32:51.924984 24130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.003s
I20260810 22:32:51.927412 24145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:51.928750 24130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:32:51.928905 24130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/wal
uuid: "5ac3344dbc9f46faa4995a94fe3d40a0"
format_stamp: "Formatted at 2026-08-10 22:32:51 on dist-test-slave-njxd"
server_key: "6e19da0daa40f1c37dee61bf8d5721ee"
server_key_iv: "c87b0de90806a62eaea3824de08076c3"
server_key_version: "encryptionkey@0"
I20260810 22:32:51.929024 24130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:51.940058 24130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:51.940758 24130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:51.940959 24130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:51.941550 24130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:51.942538 24130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:51.942620 24130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:51.942721 24130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:51.942770 24130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:51.952342 24130 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.129:37447
I20260810 22:32:51.952365 24266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.129:37447 every 8 connection(s)
I20260810 22:32:51.953262 24130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/data/info.pb
I20260810 22:32:51.953864 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 24130
I20260810 22:32:51.953997 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-0/wal/instance
I20260810 22:32:51.954372 1910 external_mini_cluster.cc:1428] Setting key 4433f027806adbe957c44b95a77d0bc4
I20260810 22:32:51.956640 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.130:0
--local_ip_for_outbound_sockets=127.1.221.130
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45129
--builtin_ntp_servers=127.1.221.148:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 22:32:51.967645 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.955286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.129:42023 (local address 127.1.221.190:45129)
0810 22:32:51.955422 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:51.955426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:51.956721 (+ 1295us) server_negotiation.cc:407] Connection header received
0810 22:32:51.958311 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:51.958315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:51.958375 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:51.958466 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:51.959387 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.959931 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.961193 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.961430 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.961553 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.967001 (+ 5448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:51.967005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:51.967010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:51.967029 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:51.967057 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:51.967069 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:51.967189 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:51.967359 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 22:32:51.967439 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:32:51.968567 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:51.955813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:45129 (local address 127.1.221.129:42023)
0810 22:32:51.956573 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:51.956625 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:51.958080 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:51.958653 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:51.958660 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:51.958745 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:51.959232 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.959246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.960081 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.960084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.961057 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:51.961068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:51.961609 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:51.961614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:51.961788 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:51.966672 (+ 4884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:32:51.966674 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:51.966732 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:32:51.966816 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:51.967194 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:51.967206 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:51.967212 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:32:51.967358 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 22:32:51.967671 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":148,"threads_started":1}
I20260810 22:32:51.969991 24267 heartbeater.cc:347] Connected to a master server at 127.1.221.190:45129
I20260810 22:32:51.970256 24267 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:51.970803 24267 heartbeater.cc:511] Master 127.1.221.190:45129 requested a full tablet report, sending...
I20260810 22:32:51.972405 24068 ts_manager.cc:194] Registered new tserver with Master: 5ac3344dbc9f46faa4995a94fe3d40a0 (127.1.221.129:37447)
I20260810 22:32:51.973563 24068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.129:42023
W20260810 22:32:52.070475 24270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:52.070760 24270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:52.070827 24270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:52.074879 24270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:52.074957 24270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:52.075047 24270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.130
I20260810 22:32:52.079813 24270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.221.130
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.130
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:52.080919 24270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:52.082485 24270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:52.092126 24278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:52.092137 24275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:52.092192 24276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:52.092355 24270 server_base.cc:1034] running on GCE node
I20260810 22:32:52.092835 24270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:52.093523 24270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:52.094725 24270 hybrid_clock.cc:648] HybridClock initialized: now 1786401172094715 us; error 26 us; skew 500 ppm
I20260810 22:32:52.096800 24270 webserver.cc:460] Webserver started at http://127.1.221.130:39307/ using document root <none> and password file <none>
I20260810 22:32:52.097385 24270 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:52.097448 24270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:52.097738 24270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:52.099926 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/data/instance:
uuid: "d8d60717d7d441bd8587569b7c99c992"
format_stamp: "Formatted at 2026-08-10 22:32:52 on dist-test-slave-njxd"
server_key: "d069bb5914f141b3f4547b93bf0f6ec8"
server_key_iv: "12cacba6393f0a02ea579feafbc604d6"
server_key_version: "encryptionkey@0"
I20260810 22:32:52.100502 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance:
uuid: "d8d60717d7d441bd8587569b7c99c992"
format_stamp: "Formatted at 2026-08-10 22:32:52 on dist-test-slave-njxd"
server_key: "d069bb5914f141b3f4547b93bf0f6ec8"
server_key_iv: "12cacba6393f0a02ea579feafbc604d6"
server_key_version: "encryptionkey@0"
I20260810 22:32:52.104417 24270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:32:52.107216 24285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:52.108474 24270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:32:52.108623 24270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/wal
uuid: "d8d60717d7d441bd8587569b7c99c992"
format_stamp: "Formatted at 2026-08-10 22:32:52 on dist-test-slave-njxd"
server_key: "d069bb5914f141b3f4547b93bf0f6ec8"
server_key_iv: "12cacba6393f0a02ea579feafbc604d6"
server_key_version: "encryptionkey@0"
I20260810 22:32:52.108736 24270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:52.122669 24270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:52.123358 24270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:52.123569 24270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:52.124133 24270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:52.125033 24270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:52.125082 24270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:52.125128 24270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:52.125150 24270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:52.134533 24270 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.130:41861
I20260810 22:32:52.134637 24412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.130:41861 every 8 connection(s)
I20260810 22:32:52.135563 24270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/data/info.pb
I20260810 22:32:52.144002 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 24270
I20260810 22:32:52.144179 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/wal/instance
I20260810 22:32:52.144547 1910 external_mini_cluster.cc:1428] Setting key fa4391733edb6b99de7e51b9952544e2
I20260810 22:32:52.145262 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.137503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:38653 (local address 127.1.221.190:45129)
0810 22:32:52.137696 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:52.137700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:52.138807 (+ 1107us) server_negotiation.cc:407] Connection header received
0810 22:32:52.140145 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:52.140148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:52.140195 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:52.140272 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:52.141166 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.141790 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.142891 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.143105 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.143372 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.144686 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:52.144689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:52.144691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:52.144704 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:52.144724 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:52.144732 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:52.144794 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:52.145013 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 22:32:52.145087 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 22:32:52.146260 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.137957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:45129 (local address 127.1.221.130:38653)
0810 22:32:52.138638 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:52.138679 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:52.139953 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:52.140440 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:52.140447 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:52.140528 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:52.141024 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.141037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.141903 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.141906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.142781 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.142790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.143408 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.143411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.143559 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.144327 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:32:52.144330 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:52.144388 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:32:52.144475 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:52.144826 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:52.144836 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:52.144842 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:32:52.144979 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 22:32:52.145252 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":137,"threads_started":1}
I20260810 22:32:52.146956 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.221.131:0
--local_ip_for_outbound_sockets=127.1.221.131
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45129
--builtin_ntp_servers=127.1.221.148:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 22:32:52.147482 24413 heartbeater.cc:347] Connected to a master server at 127.1.221.190:45129
I20260810 22:32:52.147734 24413 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:52.148289 24413 heartbeater.cc:511] Master 127.1.221.190:45129 requested a full tablet report, sending...
I20260810 22:32:52.149413 24068 ts_manager.cc:194] Registered new tserver with Master: d8d60717d7d441bd8587569b7c99c992 (127.1.221.130:41861)
I20260810 22:32:52.150048 24068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.130:38653
W20260810 22:32:52.260169 24416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:52.260488 24416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:52.260528 24416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:52.264724 24416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:52.264791 24416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:52.264878 24416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.221.131
I20260810 22:32:52.270008 24416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.221.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.221.131
--webserver_port=0
--tserver_master_addrs=127.1.221.190:45129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.221.131
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:52.271509 24416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:52.273397 24416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:52.283075 24424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:52.283069 24422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:52.283099 24421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:52.283380 24416 server_base.cc:1034] running on GCE node
I20260810 22:32:52.283763 24416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:52.284420 24416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:52.285614 24416 hybrid_clock.cc:648] HybridClock initialized: now 1786401172285560 us; error 32 us; skew 500 ppm
I20260810 22:32:52.287300 24416 webserver.cc:460] Webserver started at http://127.1.221.131:38317/ using document root <none> and password file <none>
I20260810 22:32:52.287840 24416 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:52.287904 24416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:52.288069 24416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:52.290254 24416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/data/instance:
uuid: "885e5980246049d885e2aa1932f43904"
format_stamp: "Formatted at 2026-08-10 22:32:52 on dist-test-slave-njxd"
server_key: "152664c4fe2a297d5f915fd9e4467908"
server_key_iv: "9cdca8c22b111aa5cd1aea532796d075"
server_key_version: "encryptionkey@0"
I20260810 22:32:52.290827 24416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance:
uuid: "885e5980246049d885e2aa1932f43904"
format_stamp: "Formatted at 2026-08-10 22:32:52 on dist-test-slave-njxd"
server_key: "152664c4fe2a297d5f915fd9e4467908"
server_key_iv: "9cdca8c22b111aa5cd1aea532796d075"
server_key_version: "encryptionkey@0"
I20260810 22:32:52.294589 24416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:32:52.296912 24430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:52.298169 24416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:52.298298 24416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/wal
uuid: "885e5980246049d885e2aa1932f43904"
format_stamp: "Formatted at 2026-08-10 22:32:52 on dist-test-slave-njxd"
server_key: "152664c4fe2a297d5f915fd9e4467908"
server_key_iv: "9cdca8c22b111aa5cd1aea532796d075"
server_key_version: "encryptionkey@0"
I20260810 22:32:52.298415 24416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:52.312572 24416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:52.313313 24416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:52.313503 24416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:52.314126 24416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:32:52.315150 24416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:32:52.315217 24416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:52.315284 24416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:32:52.315332 24416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:52.324054 24416 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.131:36609
I20260810 22:32:52.324090 24555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.131:36609 every 8 connection(s)
I20260810 22:32:52.325038 24416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/data/info.pb
I20260810 22:32:52.334254 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.326931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.131:54195 (local address 127.1.221.190:45129)
0810 22:32:52.327074 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:52.327078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:52.328101 (+ 1023us) server_negotiation.cc:407] Connection header received
0810 22:32:52.329460 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:52.329462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:52.329509 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:52.329614 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:52.330506 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.331031 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.332238 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.332541 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.332675 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.333842 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:52.333845 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:52.333847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:52.333858 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:52.333875 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:52.333883 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:52.333942 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:52.334083 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 22:32:52.334133 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:32:52.335168 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.327316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:45129 (local address 127.1.221.131:54195)
0810 22:32:52.327973 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:52.328014 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:52.329266 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:52.329808 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:52.329815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:52.329895 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:32:52.330380 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.330393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.331139 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.331142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.332130 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.332141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.332708 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.332711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.332854 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.333516 (+ 662us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:32:52.333518 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:32:52.333558 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:32:52.333678 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:32:52.333959 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:32:52.333967 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:32:52.333971 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 22:32:52.334067 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 22:32:52.334287 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":131,"threads_started":1}
I20260810 22:32:52.335212 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 24416
I20260810 22:32:52.335335 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/wal/instance
I20260810 22:32:52.335626 1910 external_mini_cluster.cc:1428] Setting key 3f0c4eeed400035775bb75f3ce6c5322
I20260810 22:32:52.336455 24556 heartbeater.cc:347] Connected to a master server at 127.1.221.190:45129
I20260810 22:32:52.336692 24556 heartbeater.cc:464] Registering TS with master...
I20260810 22:32:52.337288 24556 heartbeater.cc:511] Master 127.1.221.190:45129 requested a full tablet report, sending...
I20260810 22:32:52.338346 24068 ts_manager.cc:194] Registered new tserver with Master: 885e5980246049d885e2aa1932f43904 (127.1.221.131:36609)
I20260810 22:32:52.338914 24068 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.221.131:54195
I20260810 22:32:52.349565 1910 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:32:52.355630 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.351650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54570 (local address 127.1.221.190:45129)
0810 22:32:52.351836 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:52.351841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:52.351986 (+ 145us) server_negotiation.cc:407] Connection header received
0810 22:32:52.352152 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:52.352155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:52.352218 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:52.352303 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:52.352586 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.353114 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.354044 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.354260 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.354363 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.355162 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:52.355165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:52.355170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:52.355186 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:52.355206 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:52.355214 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:52.355293 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:52.355416 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 22:32:52.355473 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 22:32:52.358405 24068 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:32:52.361418 24068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:32:52.376021 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.369931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.1.221.129:37447)
0810 22:32:52.370971 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:52.370985 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:52.371011 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:32:52.371094 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:52.371099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:52.371342 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:52.371515 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:52.371816 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.373035 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.374730 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.375324 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.375454 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.375675 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:52.375739 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 22:32:52.375837 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":163,"threads_started":1}
I20260810 22:32:52.376047 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.369938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36609 (local address 127.0.0.1:52142)
0810 22:32:52.370730 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:52.370747 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:52.370839 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:52.371997 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:52.372000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:52.372020 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:52.372106 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.372112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.373559 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.373562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.374571 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.374581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.375648 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.375653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.375739 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.375761 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:32:52.375814 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 22:32:52.375884 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":123,"threads_started":1}
I20260810 22:32:52.376318 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.370173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.130:41861 (local address 127.0.0.1:56602)
0810 22:32:52.371257 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:52.371273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:52.371356 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:52.371899 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:52.371901 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:52.371919 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:52.372019 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.372028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.372992 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.372999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.374716 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.374723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.375954 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.375958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.376038 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.376071 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 22:32:52.376138 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 22:32:52.376203 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"mutex_wait_us":84,"thread_start_us":98,"threads_started":1}
I20260810 22:32:52.376655 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.371073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.1.221.131:36609)
0810 22:32:52.371503 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:52.371514 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:52.371537 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:32:52.371612 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:52.371617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:52.371830 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:52.372260 (+ 430us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:52.372388 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.373402 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.374707 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.375492 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.376133 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.376334 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:52.376390 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 22:32:52.376499 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":101,"threads_started":1}
I20260810 22:32:52.377610 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.370594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.1.221.130:41861)
0810 22:32:52.371116 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:52.371130 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:52.371460 (+ 330us) server_negotiation.cc:407] Connection header received
0810 22:32:52.371547 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:52.371552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:52.371767 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:52.371917 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:52.372143 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.372879 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.374814 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.375580 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.375691 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.375889 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:52.376504 (+ 615us) server_negotiation.cc:299] Negotiation successful
0810 22:32:52.376606 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":127,"threads_started":1}
I20260810 22:32:52.376029 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.369774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:37447 (local address 127.0.0.1:36184)
0810 22:32:52.370491 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:52.370543 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:52.370691 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:52.371524 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:52.371530 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:52.371555 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:52.371688 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.371695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.373156 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.373159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.374569 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.374580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.375455 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.375459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.375565 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.375604 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 22:32:52.375724 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 22:32:52.375848 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":97,"thread_start_us":75,"threads_started":1}
I20260810 22:32:52.380010 24339 tablet_service.cc:1467] Processing CreateTablet for tablet c50b76bfe43d40c6bd248432b5616f19 (DEFAULT_TABLE table=test-table [id=b4532b11e8e3421e9de6c7a059611026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:52.380000 24485 tablet_service.cc:1467] Processing CreateTablet for tablet c50b76bfe43d40c6bd248432b5616f19 (DEFAULT_TABLE table=test-table [id=b4532b11e8e3421e9de6c7a059611026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:52.380414 24198 tablet_service.cc:1467] Processing CreateTablet for tablet c50b76bfe43d40c6bd248432b5616f19 (DEFAULT_TABLE table=test-table [id=b4532b11e8e3421e9de6c7a059611026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:32:52.381033 24339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c50b76bfe43d40c6bd248432b5616f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:52.381039 24485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c50b76bfe43d40c6bd248432b5616f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:52.381335 24198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c50b76bfe43d40c6bd248432b5616f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:52.387276 24575 tablet_bootstrap.cc:492] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992: Bootstrap starting.
I20260810 22:32:52.388770 24576 tablet_bootstrap.cc:492] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0: Bootstrap starting.
I20260810 22:32:52.389514 24575 tablet_bootstrap.cc:654] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:52.389961 24577 tablet_bootstrap.cc:492] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904: Bootstrap starting.
I20260810 22:32:52.390321 24575 log.cc:824] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:52.390735 24576 tablet_bootstrap.cc:654] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:52.391633 24576 log.cc:824] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:52.392024 24577 tablet_bootstrap.cc:654] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:52.392637 24575 tablet_bootstrap.cc:492] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992: No bootstrap required, opened a new log
I20260810 22:32:52.392839 24575 ts_tablet_manager.cc:1397] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:32:52.392810 24577 log.cc:824] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:52.393671 24576 tablet_bootstrap.cc:492] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0: No bootstrap required, opened a new log
I20260810 22:32:52.393865 24576 ts_tablet_manager.cc:1397] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 22:32:52.394883 24577 tablet_bootstrap.cc:492] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904: No bootstrap required, opened a new log
I20260810 22:32:52.395064 24577 ts_tablet_manager.cc:1397] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 22:32:52.396497 24575 raft_consensus.cc:348] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac3344dbc9f46faa4995a94fe3d40a0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37447 } } peers { permanent_uuid: "885e5980246049d885e2aa1932f43904" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36609 } } peers { permanent_uuid: "d8d60717d7d441bd8587569b7c99c992" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41861 } }
I20260810 22:32:52.397081 24575 raft_consensus.cc:374] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:52.397153 24575 raft_consensus.cc:729] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d60717d7d441bd8587569b7c99c992, State: Initialized, Role: FOLLOWER
I20260810 22:32:52.397640 24575 consensus_queue.cc:260] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac3344dbc9f46faa4995a94fe3d40a0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37447 } } peers { permanent_uuid: "885e5980246049d885e2aa1932f43904" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36609 } } peers { permanent_uuid: "d8d60717d7d441bd8587569b7c99c992" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41861 } }
I20260810 22:32:52.398283 24413 heartbeater.cc:503] Master 127.1.221.190:45129 was elected leader, sending a full tablet report...
I20260810 22:32:52.398167 24576 raft_consensus.cc:348] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac3344dbc9f46faa4995a94fe3d40a0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37447 } } peers { permanent_uuid: "885e5980246049d885e2aa1932f43904" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36609 } } peers { permanent_uuid: "d8d60717d7d441bd8587569b7c99c992" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41861 } }
I20260810 22:32:52.398406 24576 raft_consensus.cc:374] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:52.398456 24576 raft_consensus.cc:729] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ac3344dbc9f46faa4995a94fe3d40a0, State: Initialized, Role: FOLLOWER
I20260810 22:32:52.398622 24575 ts_tablet_manager.cc:1428] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:32:52.398882 24576 consensus_queue.cc:260] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac3344dbc9f46faa4995a94fe3d40a0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37447 } } peers { permanent_uuid: "885e5980246049d885e2aa1932f43904" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36609 } } peers { permanent_uuid: "d8d60717d7d441bd8587569b7c99c992" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41861 } }
I20260810 22:32:52.399163 24577 raft_consensus.cc:348] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac3344dbc9f46faa4995a94fe3d40a0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37447 } } peers { permanent_uuid: "885e5980246049d885e2aa1932f43904" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36609 } } peers { permanent_uuid: "d8d60717d7d441bd8587569b7c99c992" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41861 } }
I20260810 22:32:52.399433 24577 raft_consensus.cc:374] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:52.399500 24577 raft_consensus.cc:729] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 885e5980246049d885e2aa1932f43904, State: Initialized, Role: FOLLOWER
I20260810 22:32:52.400008 24576 ts_tablet_manager.cc:1428] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:32:52.400008 24577 consensus_queue.cc:260] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac3344dbc9f46faa4995a94fe3d40a0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37447 } } peers { permanent_uuid: "885e5980246049d885e2aa1932f43904" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36609 } } peers { permanent_uuid: "d8d60717d7d441bd8587569b7c99c992" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41861 } }
I20260810 22:32:52.400384 24267 heartbeater.cc:503] Master 127.1.221.190:45129 was elected leader, sending a full tablet report...
I20260810 22:32:52.400622 24556 heartbeater.cc:503] Master 127.1.221.190:45129 was elected leader, sending a full tablet report...
I20260810 22:32:52.400928 24577 ts_tablet_manager.cc:1428] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260810 22:32:52.454746 24268 tablet.cc:2369] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:32:52.576113 24557 tablet.cc:2369] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:32:52.626683 24581 raft_consensus.cc:482] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:32:52.626947 24581 raft_consensus.cc:504] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac3344dbc9f46faa4995a94fe3d40a0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37447 } } peers { permanent_uuid: "885e5980246049d885e2aa1932f43904" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36609 } } peers { permanent_uuid: "d8d60717d7d441bd8587569b7c99c992" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41861 } }
I20260810 22:32:52.628226 24581 leader_election.cc:288] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ac3344dbc9f46faa4995a94fe3d40a0 (127.1.221.129:37447), 885e5980246049d885e2aa1932f43904 (127.1.221.131:36609)
I20260810 22:32:52.633104 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.628685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:41053 (local address 127.1.221.129:37447)
0810 22:32:52.628812 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:52.628819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:52.628834 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:32:52.628898 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:52.628900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:52.628941 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:52.629046 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:52.629340 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.630162 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.631760 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.632379 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.632513 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.632584 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:52.632909 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 22:32:52.632975 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:32:52.633112 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.628651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.221.130:58003 (local address 127.1.221.131:36609)
0810 22:32:52.628793 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:52.628797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:52.629036 (+ 239us) server_negotiation.cc:407] Connection header received
0810 22:32:52.629180 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:52.629182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:52.629225 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:52.629327 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:32:52.629616 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.630326 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.631760 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.632410 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.632517 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.632564 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:52.632890 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 22:32:52.632969 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:32:52.633147 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.628435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:37447 (local address 127.1.221.130:41053)
0810 22:32:52.628579 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:52.628598 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:52.628705 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:52.629060 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:52.629064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:52.629079 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:52.629207 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.629213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.630275 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.630278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.631624 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.631636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.632517 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.632521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.632682 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.632760 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 22:32:52.632851 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 22:32:52.632984 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":98}
I20260810 22:32:52.633214 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.628533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.131:36609 (local address 127.1.221.130:58003)
0810 22:32:52.628934 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:52.628945 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:32:52.629059 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:32:52.629363 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:32:52.629365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:32:52.629382 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:32:52.629481 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.629485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.630420 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.630422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.631648 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:32:52.631654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.632517 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.632560 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:32:52.632687 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.632758 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 22:32:52.632851 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 22:32:52.632985 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":116,"thread_start_us":114,"threads_started":1}
I20260810 22:32:52.633948 24220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c50b76bfe43d40c6bd248432b5616f19" candidate_uuid: "d8d60717d7d441bd8587569b7c99c992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ac3344dbc9f46faa4995a94fe3d40a0" is_pre_election: true
I20260810 22:32:52.633957 24508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c50b76bfe43d40c6bd248432b5616f19" candidate_uuid: "d8d60717d7d441bd8587569b7c99c992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885e5980246049d885e2aa1932f43904" is_pre_election: true
I20260810 22:32:52.634276 24220 raft_consensus.cc:2454] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d60717d7d441bd8587569b7c99c992 in term 0.
I20260810 22:32:52.634291 24508 raft_consensus.cc:2454] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d60717d7d441bd8587569b7c99c992 in term 0.
I20260810 22:32:52.634848 24287 leader_election.cc:302] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5ac3344dbc9f46faa4995a94fe3d40a0, 885e5980246049d885e2aa1932f43904, d8d60717d7d441bd8587569b7c99c992; no voters:
I20260810 22:32:52.635177 24581 raft_consensus.cc:2781] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:32:52.635262 24581 raft_consensus.cc:482] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:32:52.635311 24581 raft_consensus.cc:3037] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:52.636421 24581 raft_consensus.cc:504] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac3344dbc9f46faa4995a94fe3d40a0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37447 } } peers { permanent_uuid: "885e5980246049d885e2aa1932f43904" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36609 } } peers { permanent_uuid: "d8d60717d7d441bd8587569b7c99c992" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41861 } }
W20260810 22:32:52.636695 24414 tablet.cc:2369] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:32:52.636828 24581 leader_election.cc:288] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [CANDIDATE]: Term 1 election: Requested vote from peers 5ac3344dbc9f46faa4995a94fe3d40a0 (127.1.221.129:37447), 885e5980246049d885e2aa1932f43904 (127.1.221.131:36609)
I20260810 22:32:52.637239 24220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c50b76bfe43d40c6bd248432b5616f19" candidate_uuid: "d8d60717d7d441bd8587569b7c99c992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ac3344dbc9f46faa4995a94fe3d40a0"
I20260810 22:32:52.637375 24220 raft_consensus.cc:3037] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:52.637682 24508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c50b76bfe43d40c6bd248432b5616f19" candidate_uuid: "d8d60717d7d441bd8587569b7c99c992" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "885e5980246049d885e2aa1932f43904"
I20260810 22:32:52.637794 24508 raft_consensus.cc:3037] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:52.638636 24220 raft_consensus.cc:2454] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d60717d7d441bd8587569b7c99c992 in term 1.
I20260810 22:32:52.638940 24508 raft_consensus.cc:2454] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d60717d7d441bd8587569b7c99c992 in term 1.
I20260810 22:32:52.638967 24291 leader_election.cc:302] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ac3344dbc9f46faa4995a94fe3d40a0, d8d60717d7d441bd8587569b7c99c992; no voters:
I20260810 22:32:52.639222 24581 raft_consensus.cc:2781] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:52.639505 24581 raft_consensus.cc:686] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [term 1 LEADER]: Becoming Leader. State: Replica: d8d60717d7d441bd8587569b7c99c992, State: Running, Role: LEADER
I20260810 22:32:52.639865 24581 consensus_queue.cc:237] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac3344dbc9f46faa4995a94fe3d40a0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37447 } } peers { permanent_uuid: "885e5980246049d885e2aa1932f43904" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36609 } } peers { permanent_uuid: "d8d60717d7d441bd8587569b7c99c992" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41861 } }
I20260810 22:32:52.643100 24068 catalog_manager.cc:5495] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8d60717d7d441bd8587569b7c99c992 (127.1.221.130). New cstate: current_term: 1 leader_uuid: "d8d60717d7d441bd8587569b7c99c992" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ac3344dbc9f46faa4995a94fe3d40a0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "885e5980246049d885e2aa1932f43904" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d60717d7d441bd8587569b7c99c992" member_type: VOTER last_known_addr { host: "127.1.221.130" port: 41861 } health_report { overall_health: HEALTHY } } }
I20260810 22:32:52.673025 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.669701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.1.221.130:41861)
0810 22:32:52.669849 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:52.669853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:52.669866 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:32:52.669944 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:52.669946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:52.670001 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:52.670091 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:32:52.670309 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.670780 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.671664 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:52.671893 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:52.671984 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:52.672259 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:32:52.672693 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:32:52.672796 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:52.672842 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 22:32:52.672892 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:32:52.679126 24508 raft_consensus.cc:1261] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904 [term 1 FOLLOWER]: Refusing update from remote peer d8d60717d7d441bd8587569b7c99c992: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:52.679126 24220 raft_consensus.cc:1261] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0 [term 1 FOLLOWER]: Refusing update from remote peer d8d60717d7d441bd8587569b7c99c992: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:32:52.679989 24587 consensus_queue.cc:1035] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ac3344dbc9f46faa4995a94fe3d40a0" member_type: VOTER last_known_addr { host: "127.1.221.129" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:32:52.680205 24581 consensus_queue.cc:1035] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "885e5980246049d885e2aa1932f43904" member_type: VOTER last_known_addr { host: "127.1.221.131" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:32:52.696640 24067 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:52.696805 24067 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:32:52.699141 24067 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6440d4722354dcd901499a7d73587b9: Sending DeleteTablet for 3 replicas of tablet c50b76bfe43d40c6bd248432b5616f19
I20260810 22:32:52.699831 24198 tablet_service.cc:1514] Processing DeleteTablet for tablet c50b76bfe43d40c6bd248432b5616f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:52 UTC) from {username='slave'} at 127.0.0.1:36184
I20260810 22:32:52.700081 24485 tablet_service.cc:1514] Processing DeleteTablet for tablet c50b76bfe43d40c6bd248432b5616f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:52 UTC) from {username='slave'} at 127.0.0.1:52142
I20260810 22:32:52.700086 24339 tablet_service.cc:1514] Processing DeleteTablet for tablet c50b76bfe43d40c6bd248432b5616f19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:32:52 UTC) from {username='slave'} at 127.0.0.1:56602
I20260810 22:32:52.700276 24600 tablet_replica.cc:330] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0: stopping tablet replica
I20260810 22:32:52.700640 24600 raft_consensus.cc:2229] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:32:52.700847 24602 tablet_replica.cc:330] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992: stopping tablet replica
I20260810 22:32:52.700958 24600 raft_consensus.cc:2258] T c50b76bfe43d40c6bd248432b5616f19 P 5ac3344dbc9f46faa4995a94fe3d40a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:52.701103 24602 raft_consensus.cc:2229] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:32:52.701414 24602 raft_consensus.cc:2258] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:52.701483 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 24130
I20260810 22:32:52.712862 24601 tablet_replica.cc:330] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904: stopping tablet replica
I20260810 22:32:52.713243 24601 raft_consensus.cc:2229] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:32:52.713654 24601 raft_consensus.cc:2258] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:32:52.713951 24602 ts_tablet_manager.cc:1905] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:52.715312 24601 ts_tablet_manager.cc:1905] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:32:52.717657 24602 ts_tablet_manager.cc:1918] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:52.717775 24602 log.cc:1196] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-1/wal/wals/c50b76bfe43d40c6bd248432b5616f19
I20260810 22:32:52.718047 24601 ts_tablet_manager.cc:1918] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:32:52.718113 24602 ts_tablet_manager.cc:1939] T c50b76bfe43d40c6bd248432b5616f19 P d8d60717d7d441bd8587569b7c99c992: Deleting consensus metadata
I20260810 22:32:52.718245 24601 log.cc:1196] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904: Deleting WAL directory at /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786400913154531-1910-0/minicluster-data/ts-2/wal/wals/c50b76bfe43d40c6bd248432b5616f19
I20260810 22:32:52.718595 24601 ts_tablet_manager.cc:1939] T c50b76bfe43d40c6bd248432b5616f19 P 885e5980246049d885e2aa1932f43904: Deleting consensus metadata
I20260810 22:32:52.719455 24054 catalog_manager.cc:4841] TS d8d60717d7d441bd8587569b7c99c992 (127.1.221.130:41861): tablet c50b76bfe43d40c6bd248432b5616f19 (table test-table [id=b4532b11e8e3421e9de6c7a059611026]) successfully deleted
I20260810 22:32:52.719940 24053 catalog_manager.cc:4841] TS 885e5980246049d885e2aa1932f43904 (127.1.221.131:36609): tablet c50b76bfe43d40c6bd248432b5616f19 (table test-table [id=b4532b11e8e3421e9de6c7a059611026]) successfully deleted
W20260810 22:32:52.720288 24055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.221.129:37447 (error 108)
I20260810 22:32:52.721338 24568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:52.721046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.129:37447 (local address 127.0.0.1:36192)
0810 22:32:52.721187 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 22:32:52.721263 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.129:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 22:32:52.721454 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 24270
W20260810 22:32:52.724505 24055 catalog_manager.cc:4584] TS 5ac3344dbc9f46faa4995a94fe3d40a0 (127.1.221.129:37447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c50b76bfe43d40c6bd248432b5616f19: Network error: Client connection negotiation failed: client connection to 127.1.221.129:37447: connect: Connection refused (error 111)
I20260810 22:32:52.730682 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 24416
I20260810 22:32:52.738341 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 24035
2026-08-10T22:32:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1257 ms)
[----------] 6 tests from EncryptionPolicyTest (10904 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:32:52 dist-test-slave-njxd krb5kdc[24607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:32:52 dist-test-slave-njxd krb5kdc[24607](info): set up 2 sockets
Aug 10 22:32:52 dist-test-slave-njxd krb5kdc[24607](info): commencing operation
krb5kdc: starting...
W20260810 22:32:53.823089 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:32:53 dist-test-slave-njxd krb5kdc[24607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:32:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:32:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:32:53.952471 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:54.064654 24621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:54.064929 24621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:54.064970 24621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:54.068894 24621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:32:54.068953 24621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:54.068972 24621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:54.068987 24621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:32:54.069002 24621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:32:54.074326 24621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:54.075352 24621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:54.076331 24621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:54.084972 24627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:54.084991 24629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:54.084968 24626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:54.085273 24621 server_base.cc:1034] running on GCE node
I20260810 22:32:54.086009 24621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:54.087139 24621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:54.088346 24621 hybrid_clock.cc:648] HybridClock initialized: now 1786401174088315 us; error 43 us; skew 500 ppm
Aug 10 22:32:54 dist-test-slave-njxd krb5kdc[24607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:54.091125 24621 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:32:54.092083 24621 webserver.cc:460] Webserver started at http://127.0.0.1:43475/ using document root <none> and password file <none>
I20260810 22:32:54.092660 24621 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:54.092756 24621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:54.092998 24621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:54.095181 24621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "532794a1588b4797b6864b38b464b362"
format_stamp: "Formatted at 2026-08-10 22:32:54 on dist-test-slave-njxd"
server_key: "3b0327d3662a186e5479edced4665002"
server_key_iv: "08db9efecdf896eb5b0cd4deb024a784"
server_key_version: "encryptionkey@0"
I20260810 22:32:54.095747 24621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "532794a1588b4797b6864b38b464b362"
format_stamp: "Formatted at 2026-08-10 22:32:54 on dist-test-slave-njxd"
server_key: "3b0327d3662a186e5479edced4665002"
server_key_iv: "08db9efecdf896eb5b0cd4deb024a784"
server_key_version: "encryptionkey@0"
I20260810 22:32:54.099553 24621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:32:54.101944 24636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:54.103120 24621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:32:54.103281 24621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "532794a1588b4797b6864b38b464b362"
format_stamp: "Formatted at 2026-08-10 22:32:54 on dist-test-slave-njxd"
server_key: "3b0327d3662a186e5479edced4665002"
server_key_iv: "08db9efecdf896eb5b0cd4deb024a784"
server_key_version: "encryptionkey@0"
I20260810 22:32:54.103394 24621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:54.135077 24621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:54.138152 24621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:54.138374 24621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:54.146090 24621 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43091
I20260810 22:32:54.146118 24694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43091 every 8 connection(s)
I20260810 22:32:54.147078 24621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:32:54.149756 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 24621
I20260810 22:32:54.149896 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:32:54.149978 24695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:54.150213 1910 external_mini_cluster.cc:1428] Setting key 11290df94c0032447e53c7e4fe4c7a28
I20260810 22:32:54.156500 24695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362: Bootstrap starting.
I20260810 22:32:54.158905 24695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362: Neither blocks nor log segments found. Creating new log.
Aug 10 22:32:54 dist-test-slave-njxd krb5kdc[24607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 22:32:54.159709 24695 log.cc:824] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:54.161633 24695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362: No bootstrap required, opened a new log
I20260810 22:32:54.164732 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:54.151756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45841 (local address 127.0.0.1:43091)
0810 22:32:54.152222 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:54.152237 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:54.152274 (+ 37us) server_negotiation.cc:407] Connection header received
0810 22:32:54.152893 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:54.152910 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:54.153244 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:54.153586 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:54.154450 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:54.155477 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:54.156388 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:54.156934 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:54.157109 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:54.159857 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:54.159865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:54.159885 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:54.159923 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:54.161784 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:54.162323 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:54.162329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:54.162342 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:54.162463 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:54.162833 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:54.162837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:54.162838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:54.163225 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:54.163443 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:54.163789 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 22:32:54.164005 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":141,"threads_started":1}
I20260810 22:32:54.165412 24695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532794a1588b4797b6864b38b464b362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43091 } }
I20260810 22:32:54.165684 24695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:54.165786 24695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 532794a1588b4797b6864b38b464b362, State: Initialized, Role: FOLLOWER
I20260810 22:32:54.166278 24695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532794a1588b4797b6864b38b464b362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43091 } }
I20260810 22:32:54.166438 24695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:32:54.166502 24695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:32:54.166602 24695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:54.167714 24695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532794a1588b4797b6864b38b464b362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43091 } }
I20260810 22:32:54.167918 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:32:54.168105 24695 leader_election.cc:302] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 532794a1588b4797b6864b38b464b362; no voters:
I20260810 22:32:54.168468 24695 leader_election.cc:288] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:32:54.168526 24700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:54.168735 24700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [term 1 LEADER]: Becoming Leader. State: Replica: 532794a1588b4797b6864b38b464b362, State: Running, Role: LEADER
I20260810 22:32:54.169044 24700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532794a1588b4797b6864b38b464b362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43091 } }
I20260810 22:32:54.169776 24695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:32:54.170831 24702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 532794a1588b4797b6864b38b464b362. Latest consensus state: current_term: 1 leader_uuid: "532794a1588b4797b6864b38b464b362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532794a1588b4797b6864b38b464b362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43091 } } }
I20260810 22:32:54.170975 24702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:54.171322 24701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "532794a1588b4797b6864b38b464b362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "532794a1588b4797b6864b38b464b362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43091 } } }
I20260810 22:32:54.171458 24701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:54.171383 24717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:32:54.175459 24717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:32:54.180344 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:54.170008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50647 (local address 127.0.0.1:43091)
0810 22:32:54.170111 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:54.170115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:54.172299 (+ 2184us) server_negotiation.cc:407] Connection header received
0810 22:32:54.172465 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:54.172468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:54.172518 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:54.172600 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:54.173422 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:54.173994 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:54.174802 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:54.175028 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:54.175140 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:54.176598 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:54.176602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:54.176604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:54.176637 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:54.178301 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:54.178880 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:54.178883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:54.178885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:54.178970 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:54.179355 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:54.179358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:54.179360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:54.179572 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:54.179749 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:54.180078 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 22:32:54.180179 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 22:32:54.181658 24717 catalog_manager.cc:1347] Generated new cluster ID: 69fb89ca133e457383be1957453f62d0
I20260810 22:32:54.181722 24717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:32:54.193100 24717 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:32:54.194064 24717 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:32:54.201166 24717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 532794a1588b4797b6864b38b464b362: Generated new TSK 0
I20260810 22:32:54.201905 24717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:32:54.208485 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 24621
W20260810 22:32:54.217676 24605 connection.cc:535] client connection to 127.0.0.1:43091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43091): Connection reset by peer (error 104)
2026-08-10T22:32:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:32:54 dist-test-slave-njxd krb5kdc[24729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:32:54 dist-test-slave-njxd krb5kdc[24729](info): set up 2 sockets
Aug 10 22:32:54 dist-test-slave-njxd krb5kdc[24729](info): commencing operation
krb5kdc: starting...
W20260810 22:32:55.299387 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:32:55 dist-test-slave-njxd krb5kdc[24729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:32:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:32:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:32:55.428381 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46773
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:55.543195 24742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:55.543479 24742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:55.543553 24742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:55.547698 24742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:32:55.547760 24742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:55.547780 24742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:55.547796 24742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:32:55.547812 24742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:32:55.553098 24742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46773
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:55.554275 24742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:55.555398 24742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:55.564076 24749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:55.564076 24750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:55.564239 24742 server_base.cc:1034] running on GCE node
W20260810 22:32:55.564428 24752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:55.565071 24742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:55.566212 24742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:55.567380 24742 hybrid_clock.cc:648] HybridClock initialized: now 1786401175567352 us; error 40 us; skew 500 ppm
Aug 10 22:32:55 dist-test-slave-njxd krb5kdc[24729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:55.570173 24742 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:32:55.571025 24742 webserver.cc:460] Webserver started at http://127.0.0.1:43353/ using document root <none> and password file <none>
I20260810 22:32:55.571590 24742 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:55.571681 24742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:55.571915 24742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:55.574079 24742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "559570227b574ecfab21af995f06ea9c"
format_stamp: "Formatted at 2026-08-10 22:32:55 on dist-test-slave-njxd"
server_key: "a83834601d5d2e6de445128b6e91cb5a"
server_key_iv: "7f6305f0874a3a19e679d9cd2d14fd70"
server_key_version: "encryptionkey@0"
I20260810 22:32:55.574685 24742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "559570227b574ecfab21af995f06ea9c"
format_stamp: "Formatted at 2026-08-10 22:32:55 on dist-test-slave-njxd"
server_key: "a83834601d5d2e6de445128b6e91cb5a"
server_key_iv: "7f6305f0874a3a19e679d9cd2d14fd70"
server_key_version: "encryptionkey@0"
I20260810 22:32:55.578480 24742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:32:55.580843 24761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:55.582121 24742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:32:55.582261 24742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "559570227b574ecfab21af995f06ea9c"
format_stamp: "Formatted at 2026-08-10 22:32:55 on dist-test-slave-njxd"
server_key: "a83834601d5d2e6de445128b6e91cb5a"
server_key_iv: "7f6305f0874a3a19e679d9cd2d14fd70"
server_key_version: "encryptionkey@0"
I20260810 22:32:55.582374 24742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:55.612598 24742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:55.615625 24742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:55.615866 24742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:55.623251 24742 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46773
I20260810 22:32:55.623283 24819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46773 every 8 connection(s)
I20260810 22:32:55.624212 24742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:32:55.625689 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 24742
I20260810 22:32:55.625844 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:32:55.626159 1910 external_mini_cluster.cc:1428] Setting key 82121e4a37770447ce6f38a144bbe170
I20260810 22:32:55.627439 24820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:55.633185 24820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c: Bootstrap starting.
I20260810 22:32:55.635340 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:55.627609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49257 (local address 127.0.0.1:46773)
0810 22:32:55.628053 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:55.628068 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:55.628104 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:32:55.628737 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:55.628753 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:55.629087 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:55.629423 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:55.629835 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:55.630780 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:55.631828 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:55.632395 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:55.632555 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:55.633606 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:55.633617 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:55.633634 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:55.634036 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:55.634091 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:55.634114 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:55.634260 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:55.634455 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 22:32:55.634611 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":149,"threads_started":1}
I20260810 22:32:55.635668 24820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:55.636488 24820 log.cc:824] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:55.638517 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:32:55.638928 24820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c: No bootstrap required, opened a new log
I20260810 22:32:55.642895 24820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559570227b574ecfab21af995f06ea9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46773 } }
I20260810 22:32:55.643158 24820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:55.643242 24820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559570227b574ecfab21af995f06ea9c, State: Initialized, Role: FOLLOWER
I20260810 22:32:55.643747 24820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559570227b574ecfab21af995f06ea9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46773 } }
I20260810 22:32:55.643930 24820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:32:55.644016 24820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:32:55.644153 24820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:55.644662 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:55.640591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46493 (local address 127.0.0.1:46773)
0810 22:32:55.640736 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:55.640742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:55.640911 (+ 169us) server_negotiation.cc:407] Connection header received
0810 22:32:55.641080 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:55.641088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:55.641125 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:55.641209 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:55.641485 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:55.642069 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:55.642897 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:55.643128 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:55.643247 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:55.644197 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:55.644200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:55.644205 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:55.644221 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:55.644253 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:55.644268 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:55.644371 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:55.644489 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 22:32:55.644528 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 22:32:55.645368 24820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559570227b574ecfab21af995f06ea9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46773 } }
I20260810 22:32:55.645759 24820 leader_election.cc:302] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 559570227b574ecfab21af995f06ea9c; no voters:
I20260810 22:32:55.646108 24820 leader_election.cc:288] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:32:55.646226 24833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:55.646497 24833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [term 1 LEADER]: Becoming Leader. State: Replica: 559570227b574ecfab21af995f06ea9c, State: Running, Role: LEADER
I20260810 22:32:55.646884 24833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559570227b574ecfab21af995f06ea9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46773 } }
I20260810 22:32:55.647521 24820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:32:55.648666 24835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 559570227b574ecfab21af995f06ea9c. Latest consensus state: current_term: 1 leader_uuid: "559570227b574ecfab21af995f06ea9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559570227b574ecfab21af995f06ea9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46773 } } }
I20260810 22:32:55.648800 24835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:55.649214 24843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:32:55.649161 24834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "559570227b574ecfab21af995f06ea9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "559570227b574ecfab21af995f06ea9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46773 } } }
I20260810 22:32:55.649968 24834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:55.652204 24843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:32:55.658134 24843 catalog_manager.cc:1347] Generated new cluster ID: 2638c90a62214d3781e5681a0cc9fecf
I20260810 22:32:55.658223 24843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:32:55.671470 24843 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:32:55.672541 24843 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:32:55.677534 24843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 559570227b574ecfab21af995f06ea9c: Generated new TSK 0
I20260810 22:32:55.678268 24843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:32:55.683205 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 24742
W20260810 22:32:55.690946 24727 connection.cc:535] client connection to 127.0.0.1:46773 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46773): Connection reset by peer (error 104)
2026-08-10T22:32:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:32:55 dist-test-slave-njxd krb5kdc[24857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:32:55 dist-test-slave-njxd krb5kdc[24857](info): set up 2 sockets
Aug 10 22:32:55 dist-test-slave-njxd krb5kdc[24857](info): commencing operation
krb5kdc: starting...
W20260810 22:32:56.770802 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:32:56 dist-test-slave-njxd krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:32:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:32:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:32:56.901117 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:57.015949 24871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:57.016223 24871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:57.016268 24871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:57.020308 24871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:32:57.020375 24871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:57.020393 24871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:57.020411 24871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:32:57.020424 24871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:32:57.026068 24871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41817
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:57.027184 24871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:57.028249 24871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:57.036906 24880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:57.036906 24878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:57.037084 24871 server_base.cc:1034] running on GCE node
W20260810 22:32:57.036964 24876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:57.037766 24871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:57.038853 24871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:57.040047 24871 hybrid_clock.cc:648] HybridClock initialized: now 1786401177040029 us; error 32 us; skew 500 ppm
Aug 10 22:32:57 dist-test-slave-njxd krb5kdc[24857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:57.046412 24871 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:32:57.047371 24871 webserver.cc:460] Webserver started at http://127.0.0.1:46763/ using document root <none> and password file <none>
I20260810 22:32:57.047941 24871 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:57.048035 24871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:57.048274 24871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:57.050359 24871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "0c5141f2c95042789a801e07dcb337d0"
format_stamp: "Formatted at 2026-08-10 22:32:57 on dist-test-slave-njxd"
server_key: "73a63970b7bd4405ec68e8fd04201949"
server_key_iv: "8a169654717540d21a68ccff77588b80"
server_key_version: "encryptionkey@0"
I20260810 22:32:57.050931 24871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "0c5141f2c95042789a801e07dcb337d0"
format_stamp: "Formatted at 2026-08-10 22:32:57 on dist-test-slave-njxd"
server_key: "73a63970b7bd4405ec68e8fd04201949"
server_key_iv: "8a169654717540d21a68ccff77588b80"
server_key_version: "encryptionkey@0"
I20260810 22:32:57.054684 24871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:32:57.057018 24888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:57.058234 24871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:32:57.058377 24871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "0c5141f2c95042789a801e07dcb337d0"
format_stamp: "Formatted at 2026-08-10 22:32:57 on dist-test-slave-njxd"
server_key: "73a63970b7bd4405ec68e8fd04201949"
server_key_iv: "8a169654717540d21a68ccff77588b80"
server_key_version: "encryptionkey@0"
I20260810 22:32:57.058487 24871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:57.063035 24871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:57.065863 24871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:57.066059 24871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:57.074467 24871 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41817
I20260810 22:32:57.074481 24944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41817 every 8 connection(s)
I20260810 22:32:57.075779 24871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:32:57.077796 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 24871
I20260810 22:32:57.077934 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:32:57.078310 1910 external_mini_cluster.cc:1428] Setting key 598c135a9d976e2fc642c2d72e0a3363
I20260810 22:32:57.079620 24946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:57.083725 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:57.079901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42485 (local address 127.0.0.1:41817)
0810 22:32:57.080364 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:57.080383 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:57.080430 (+ 47us) server_negotiation.cc:407] Connection header received
0810 22:32:57.081052 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:57.081067 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:57.081335 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:57.081688 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:57.082506 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:57.082510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:57.082522 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:57.082531 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:57.082579 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:57.082598 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:57.082706 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:57.082827 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 22:32:57.082990 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":158,"threads_started":1}
I20260810 22:32:57.085973 24946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0: Bootstrap starting.
I20260810 22:32:57.086571 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:32:57.088301 24946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:57.089074 24946 log.cc:824] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:57.090441 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:57.088916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45735 (local address 127.0.0.1:41817)
0810 22:32:57.089054 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:57.089059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:57.089076 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:32:57.089114 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:57.089116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:57.089152 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:57.089235 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:57.090083 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:57.090086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:57.090089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:32:57.090100 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:57.090125 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:32:57.090138 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:57.090233 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:57.090297 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 22:32:57.090332 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:32:57.091188 24946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0: No bootstrap required, opened a new log
I20260810 22:32:57.094753 24946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5141f2c95042789a801e07dcb337d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41817 } }
I20260810 22:32:57.094972 24946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:57.095007 24946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c5141f2c95042789a801e07dcb337d0, State: Initialized, Role: FOLLOWER
I20260810 22:32:57.095355 24946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5141f2c95042789a801e07dcb337d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41817 } }
I20260810 22:32:57.095486 24946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:32:57.095559 24946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:32:57.095633 24946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:57.096459 24946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5141f2c95042789a801e07dcb337d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41817 } }
I20260810 22:32:57.096822 24946 leader_election.cc:302] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c5141f2c95042789a801e07dcb337d0; no voters:
I20260810 22:32:57.097189 24946 leader_election.cc:288] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:32:57.097307 24956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:57.097489 24956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [term 1 LEADER]: Becoming Leader. State: Replica: 0c5141f2c95042789a801e07dcb337d0, State: Running, Role: LEADER
I20260810 22:32:57.097810 24956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5141f2c95042789a801e07dcb337d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41817 } }
I20260810 22:32:57.098544 24946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:32:57.099684 24958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c5141f2c95042789a801e07dcb337d0. Latest consensus state: current_term: 1 leader_uuid: "0c5141f2c95042789a801e07dcb337d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5141f2c95042789a801e07dcb337d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41817 } } }
I20260810 22:32:57.099710 24957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c5141f2c95042789a801e07dcb337d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c5141f2c95042789a801e07dcb337d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41817 } } }
I20260810 22:32:57.099825 24957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:57.099825 24958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:57.100345 24966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:32:57.103574 24966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:32:57.109326 24966 catalog_manager.cc:1347] Generated new cluster ID: 63420cc371b84c53837bc20abee333bb
I20260810 22:32:57.109412 24966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:32:57.127138 24966 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:32:57.127483 24966 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:32:57.133642 24966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c5141f2c95042789a801e07dcb337d0: Generated new TSK 0
I20260810 22:32:57.134406 24966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:32:57.137431 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 24871
2026-08-10T22:32:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:32:57 dist-test-slave-njxd krb5kdc[24981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:32:57 dist-test-slave-njxd krb5kdc[24981](info): set up 2 sockets
Aug 10 22:32:57 dist-test-slave-njxd krb5kdc[24981](info): commencing operation
krb5kdc: starting...
W20260810 22:32:58.229717 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:32:58 dist-test-slave-njxd krb5kdc[24981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:32:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:32:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:32:58.357646 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:58.472772 24995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:58.473083 24995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:58.473129 24995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:58.477185 24995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:32:58.477255 24995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:58.477274 24995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:58.477291 24995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:32:58.477308 24995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:32:58.482637 24995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:58.483712 24995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:58.484841 24995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:58.493635 25001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:58.493685 25003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:58.493654 25000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:58.493993 24995 server_base.cc:1034] running on GCE node
I20260810 22:32:58.494644 24995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:58.495806 24995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:58.497004 24995 hybrid_clock.cc:648] HybridClock initialized: now 1786401178496976 us; error 50 us; skew 500 ppm
Aug 10 22:32:58 dist-test-slave-njxd krb5kdc[24981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:58.499850 24995 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:32:58.500834 24995 webserver.cc:460] Webserver started at http://127.0.0.1:35981/ using document root <none> and password file <none>
I20260810 22:32:58.501390 24995 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:58.501451 24995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:58.501682 24995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:58.503769 24995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "ad2f63ead5794ff5af5b1aa29c3c5d59"
format_stamp: "Formatted at 2026-08-10 22:32:58 on dist-test-slave-njxd"
server_key: "c0ce19b3305b149086a76ccce2879868"
server_key_iv: "9492a67de3b758742ffb0140565ca5a3"
server_key_version: "encryptionkey@0"
I20260810 22:32:58.504372 24995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "ad2f63ead5794ff5af5b1aa29c3c5d59"
format_stamp: "Formatted at 2026-08-10 22:32:58 on dist-test-slave-njxd"
server_key: "c0ce19b3305b149086a76ccce2879868"
server_key_iv: "9492a67de3b758742ffb0140565ca5a3"
server_key_version: "encryptionkey@0"
I20260810 22:32:58.508158 24995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:32:58.510650 25012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:58.511934 24995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:32:58.512084 24995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "ad2f63ead5794ff5af5b1aa29c3c5d59"
format_stamp: "Formatted at 2026-08-10 22:32:58 on dist-test-slave-njxd"
server_key: "c0ce19b3305b149086a76ccce2879868"
server_key_iv: "9492a67de3b758742ffb0140565ca5a3"
server_key_version: "encryptionkey@0"
I20260810 22:32:58.512199 24995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:32:58.536376 24995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:32:58.539330 24995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:32:58.539537 24995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:32:58.546900 24995 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37435
I20260810 22:32:58.546926 25065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37435 every 8 connection(s)
I20260810 22:32:58.547863 24995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:32:58.550639 25066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:32:58.555169 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 24995
I20260810 22:32:58.555316 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:32:58.555596 1910 external_mini_cluster.cc:1428] Setting key eae433991a713ebaac8d46e6c8adb242
I20260810 22:32:58.556473 25066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59: Bootstrap starting.
I20260810 22:32:58.559069 25066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59: Neither blocks nor log segments found. Creating new log.
I20260810 22:32:58.559877 25066 log.cc:824] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59: Log is configured to *not* fsync() on all Append() calls
I20260810 22:32:58.561837 25066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59: No bootstrap required, opened a new log
Aug 10 22:32:58 dist-test-slave-njxd krb5kdc[24981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 22:32:58.565701 25066 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2f63ead5794ff5af5b1aa29c3c5d59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37435 } }
I20260810 22:32:58.565954 25066 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:32:58.566018 25066 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2f63ead5794ff5af5b1aa29c3c5d59, State: Initialized, Role: FOLLOWER
I20260810 22:32:58.566457 25066 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2f63ead5794ff5af5b1aa29c3c5d59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37435 } }
I20260810 22:32:58.566625 25066 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:32:58.566717 25066 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:32:58.566845 25066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:32:58.567742 25066 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2f63ead5794ff5af5b1aa29c3c5d59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37435 } }
I20260810 22:32:58.568074 25066 leader_election.cc:302] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad2f63ead5794ff5af5b1aa29c3c5d59; no voters:
I20260810 22:32:58.568373 25066 leader_election.cc:288] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:32:58.568483 25071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:32:58.568760 25071 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [term 1 LEADER]: Becoming Leader. State: Replica: ad2f63ead5794ff5af5b1aa29c3c5d59, State: Running, Role: LEADER
I20260810 22:32:58.569135 25071 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2f63ead5794ff5af5b1aa29c3c5d59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37435 } }
I20260810 22:32:58.569792 25066 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:32:58.570405 25073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad2f63ead5794ff5af5b1aa29c3c5d59. Latest consensus state: current_term: 1 leader_uuid: "ad2f63ead5794ff5af5b1aa29c3c5d59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2f63ead5794ff5af5b1aa29c3c5d59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37435 } } }
I20260810 22:32:58.570519 25073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:58.570792 25069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:58.557026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60877 (local address 127.0.0.1:37435)
0810 22:32:58.557488 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:58.557504 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:58.557541 (+ 37us) server_negotiation.cc:407] Connection header received
0810 22:32:58.558275 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:58.558291 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:58.558643 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:58.559000 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:58.559829 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:58.560875 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:58.561797 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:58.562324 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:58.562499 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:58.565001 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:58.565006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:58.565020 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:58.565046 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:58.567107 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:58.567540 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:58.567545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:58.567551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:58.567674 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:58.567993 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:58.567997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:58.567998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:58.568492 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:58.568689 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:58.569451 (+ 762us) server_negotiation.cc:299] Negotiation successful
0810 22:32:58.569778 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":160,"threads_started":1}
I20260810 22:32:58.571188 25072 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad2f63ead5794ff5af5b1aa29c3c5d59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2f63ead5794ff5af5b1aa29c3c5d59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37435 } } }
I20260810 22:32:58.571272 25072 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59 [sys.catalog]: This master's current role is: LEADER
I20260810 22:32:58.573969 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 22:32:58.575347 25087 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:32:58.575469 25087 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:32:58.575572 25078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:32:58.576465 25078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:32:58.584303 25078 catalog_manager.cc:1347] Generated new cluster ID: a735fc39fdaf4cbaa43f63301cf7f475
I20260810 22:32:58.584306 25069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:32:58.576216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34877 (local address 127.0.0.1:37435)
0810 22:32:58.576315 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 22:32:58.576319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:32:58.576469 (+ 150us) server_negotiation.cc:407] Connection header received
0810 22:32:58.576580 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:32:58.576583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:32:58.576628 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:32:58.576714 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:32:58.577556 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:58.578328 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:58.579142 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:32:58.579391 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:32:58.579502 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:32:58.580920 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:32:58.580924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:32:58.580927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:32:58.580952 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:32:58.582655 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:58.583092 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:58.583096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:58.583098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:58.583184 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:32:58.583463 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:32:58.583468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:32:58.583470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:32:58.583651 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:32:58.583787 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 22:32:58.584019 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 22:32:58.584149 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 22:32:58.584386 25078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:32:58.596498 25078 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:32:58.597553 25078 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:32:58.605968 25078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad2f63ead5794ff5af5b1aa29c3c5d59: Generated new TSK 0
I20260810 22:32:58.606689 25078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:32:58.613426 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 24995
2026-08-10T22:32:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:32:58 dist-test-slave-njxd krb5kdc[25099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:32:58 dist-test-slave-njxd krb5kdc[25099](info): set up 2 sockets
Aug 10 22:32:58 dist-test-slave-njxd krb5kdc[25099](info): commencing operation
krb5kdc: starting...
W20260810 22:32:59.706359 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:32:59 dist-test-slave-njxd krb5kdc[25099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:32:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:32:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:32:59.840112 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:32:59.954761 25112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:32:59.955044 25112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:32:59.955117 25112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:32:59.959272 25112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:32:59.959342 25112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:32:59.959362 25112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:32:59.959420 25112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:32:59.959455 25112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:32:59.964845 25112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:32:59.966019 25112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:32:59.967173 25112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:32:59.975896 25119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:59.975893 25121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:32:59.975909 25118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:32:59.976169 25112 server_base.cc:1034] running on GCE node
I20260810 22:32:59.976933 25112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:32:59.978199 25112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:32:59.979394 25112 hybrid_clock.cc:648] HybridClock initialized: now 1786401179979380 us; error 27 us; skew 500 ppm
Aug 10 22:32:59 dist-test-slave-njxd krb5kdc[25099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:32:59.982249 25112 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:32:59.983132 25112 webserver.cc:460] Webserver started at http://127.0.0.1:45483/ using document root <none> and password file <none>
I20260810 22:32:59.983693 25112 fs_manager.cc:359] Metadata directory not provided
I20260810 22:32:59.983753 25112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:32:59.983999 25112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:32:59.986133 25112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "08563ae91c4a4a5bbf6c7c194bde10b8"
format_stamp: "Formatted at 2026-08-10 22:32:59 on dist-test-slave-njxd"
server_key: "08043678e4bd1dd548b2a484dfdb92bd"
server_key_iv: "4d57293641811778afeb06252fc8aa7e"
server_key_version: "encryptionkey@0"
I20260810 22:32:59.986718 25112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "08563ae91c4a4a5bbf6c7c194bde10b8"
format_stamp: "Formatted at 2026-08-10 22:32:59 on dist-test-slave-njxd"
server_key: "08043678e4bd1dd548b2a484dfdb92bd"
server_key_iv: "4d57293641811778afeb06252fc8aa7e"
server_key_version: "encryptionkey@0"
I20260810 22:32:59.990355 25112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:32:59.992759 25129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:32:59.994037 25112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:32:59.994160 25112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "08563ae91c4a4a5bbf6c7c194bde10b8"
format_stamp: "Formatted at 2026-08-10 22:32:59 on dist-test-slave-njxd"
server_key: "08043678e4bd1dd548b2a484dfdb92bd"
server_key_iv: "4d57293641811778afeb06252fc8aa7e"
server_key_version: "encryptionkey@0"
I20260810 22:32:59.994256 25112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:00.019533 25112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:00.022475 25112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:00.022698 25112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:00.030151 25112 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38371
I20260810 22:33:00.030190 25187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38371 every 8 connection(s)
I20260810 22:33:00.031149 25112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:00.033903 25188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:00.037861 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 25112
I20260810 22:33:00.038022 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:00.038326 1910 external_mini_cluster.cc:1428] Setting key 222e1c52ce9737ff62988eaef5f1b897
I20260810 22:33:00.039863 25188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8: Bootstrap starting.
I20260810 22:33:00.042227 25188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:00.043064 25188 log.cc:824] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:00.045006 25188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8: No bootstrap required, opened a new log
I20260810 22:33:00.047546 25191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:00.039767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57283 (local address 127.0.0.1:38371)
0810 22:33:00.040218 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:00.040233 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:00.040268 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:33:00.040902 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:00.040918 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:00.041281 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:00.041645 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:00.042083 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:00.043264 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:00.044189 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:00.044691 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:00.044876 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:00.045858 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:00.045863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:00.045877 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:00.046280 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:00.046337 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:00.046360 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:00.046478 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:00.046603 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 22:33:00.046828 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":135,"threads_started":1}
I20260810 22:33:00.048959 25188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08563ae91c4a4a5bbf6c7c194bde10b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38371 } }
I20260810 22:33:00.049213 25188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:00.049290 25188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08563ae91c4a4a5bbf6c7c194bde10b8, State: Initialized, Role: FOLLOWER
I20260810 22:33:00.049872 25188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08563ae91c4a4a5bbf6c7c194bde10b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38371 } }
I20260810 22:33:00.050047 25188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:00.050113 25188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:00.050225 25188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:00.050642 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:00.051465 25188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08563ae91c4a4a5bbf6c7c194bde10b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38371 } }
I20260810 22:33:00.051836 25188 leader_election.cc:302] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08563ae91c4a4a5bbf6c7c194bde10b8; no voters:
I20260810 22:33:00.052201 25188 leader_election.cc:288] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:00.052294 25196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:00.052525 25196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [term 1 LEADER]: Becoming Leader. State: Replica: 08563ae91c4a4a5bbf6c7c194bde10b8, State: Running, Role: LEADER
I20260810 22:33:00.052855 25196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08563ae91c4a4a5bbf6c7c194bde10b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38371 } }
I20260810 22:33:00.053531 25188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:00.054965 25199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08563ae91c4a4a5bbf6c7c194bde10b8. Latest consensus state: current_term: 1 leader_uuid: "08563ae91c4a4a5bbf6c7c194bde10b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08563ae91c4a4a5bbf6c7c194bde10b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38371 } } }
I20260810 22:33:00.055105 25199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:00.055212 25198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08563ae91c4a4a5bbf6c7c194bde10b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08563ae91c4a4a5bbf6c7c194bde10b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38371 } } }
I20260810 22:33:00.055368 25198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:00.055691 25209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:00.057864 25191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:00.053266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44919 (local address 127.0.0.1:38371)
0810 22:33:00.053362 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:00.053366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:00.053379 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:33:00.053526 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:00.053529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:00.053604 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:00.053687 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:00.054088 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:00.054861 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:00.056023 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:00.056358 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:00.056464 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:00.057369 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:00.057373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:00.057376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:00.057389 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:00.057417 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:00.057429 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:00.057508 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:00.057661 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 22:33:00.057730 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 22:33:00.059405 25209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:00.064527 25209 catalog_manager.cc:1347] Generated new cluster ID: 365ea49ca80643ca81b49852b50229b2
I20260810 22:33:00.064597 25209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:00.079380 25209 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:00.080494 25209 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:00.086892 25209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08563ae91c4a4a5bbf6c7c194bde10b8: Generated new TSK 0
I20260810 22:33:00.087535 25209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:00.095649 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 25112
2026-08-10T22:33:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1484 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:00 dist-test-slave-njxd krb5kdc[25222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:00 dist-test-slave-njxd krb5kdc[25222](info): set up 2 sockets
Aug 10 22:33:00 dist-test-slave-njxd krb5kdc[25222](info): commencing operation
krb5kdc: starting...
W20260810 22:33:01.194265 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:01 dist-test-slave-njxd krb5kdc[25222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:33:01.327924 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:01.440514 25237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:01.440834 25237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:01.440879 25237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:01.445099 25237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:01.445176 25237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:01.445196 25237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:01.445214 25237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:01.445258 25237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:01.450904 25237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:01.452085 25237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:01.453181 25237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:01.462039 25242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:01.462049 25245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:01.462049 25243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:01.462391 25237 server_base.cc:1034] running on GCE node
I20260810 22:33:01.463017 25237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:01.464293 25237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:01.465493 25237 hybrid_clock.cc:648] HybridClock initialized: now 1786401181465474 us; error 36 us; skew 500 ppm
Aug 10 22:33:01 dist-test-slave-njxd krb5kdc[25222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:01.468459 25237 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:33:01.469429 25237 webserver.cc:460] Webserver started at http://127.0.0.1:43717/ using document root <none> and password file <none>
I20260810 22:33:01.470082 25237 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:01.470184 25237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:01.470438 25237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:01.472621 25237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "20f7dbcfaac64b28900bd39440942f9c"
format_stamp: "Formatted at 2026-08-10 22:33:01 on dist-test-slave-njxd"
server_key: "3edea8927692aa7081d3accfc8a866f1"
server_key_iv: "0716033e9a61a77bc933f0f8618054a1"
server_key_version: "encryptionkey@0"
I20260810 22:33:01.473241 25237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "20f7dbcfaac64b28900bd39440942f9c"
format_stamp: "Formatted at 2026-08-10 22:33:01 on dist-test-slave-njxd"
server_key: "3edea8927692aa7081d3accfc8a866f1"
server_key_iv: "0716033e9a61a77bc933f0f8618054a1"
server_key_version: "encryptionkey@0"
I20260810 22:33:01.477442 25237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 22:33:01.480250 25252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:01.481686 25237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:33:01.481884 25237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "20f7dbcfaac64b28900bd39440942f9c"
format_stamp: "Formatted at 2026-08-10 22:33:01 on dist-test-slave-njxd"
server_key: "3edea8927692aa7081d3accfc8a866f1"
server_key_iv: "0716033e9a61a77bc933f0f8618054a1"
server_key_version: "encryptionkey@0"
I20260810 22:33:01.482029 25237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:01.486940 25237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:01.490339 25237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:01.490553 25237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:01.497938 25237 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42981
I20260810 22:33:01.497956 25313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42981 every 8 connection(s)
I20260810 22:33:01.498925 25237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:01.501874 25314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:01.504333 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 25237
I20260810 22:33:01.504513 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:01.504822 1910 external_mini_cluster.cc:1428] Setting key 14f482b85cb8805aabf986e5e2824cdb
I20260810 22:33:01.508388 25314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c: Bootstrap starting.
I20260810 22:33:01.510648 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:01.506352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57611 (local address 127.0.0.1:42981)
0810 22:33:01.506800 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:01.506825 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:01.506869 (+ 44us) server_negotiation.cc:407] Connection header received
0810 22:33:01.507682 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:01.507708 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:01.508073 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:01.508446 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:01.509190 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:01.509196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:01.509212 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:01.509224 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:01.509288 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:01.509314 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:01.509452 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:01.509525 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 22:33:01.509730 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":121,"threads_started":1}
I20260810 22:33:01.511335 25314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:01.512151 25314 log.cc:824] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:01.513731 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:01.514402 25314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c: No bootstrap required, opened a new log
I20260810 22:33:01.517529 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:01.515847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59767 (local address 127.0.0.1:42981)
0810 22:33:01.515980 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:01.515985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:01.516184 (+ 199us) server_negotiation.cc:407] Connection header received
0810 22:33:01.516318 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:01.516321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:01.516352 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:01.516417 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:01.517147 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:01.517149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:01.517151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:01.517161 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:01.517186 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:01.517196 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:01.517265 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:01.517370 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 22:33:01.517399 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:33:01.519114 25314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f7dbcfaac64b28900bd39440942f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42981 } }
I20260810 22:33:01.519382 25314 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:01.519462 25314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f7dbcfaac64b28900bd39440942f9c, State: Initialized, Role: FOLLOWER
I20260810 22:33:01.519935 25314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f7dbcfaac64b28900bd39440942f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42981 } }
I20260810 22:33:01.520092 25314 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:01.520164 25314 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:01.520273 25314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:01.521356 25314 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f7dbcfaac64b28900bd39440942f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42981 } }
I20260810 22:33:01.521757 25314 leader_election.cc:302] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20f7dbcfaac64b28900bd39440942f9c; no voters:
I20260810 22:33:01.522128 25314 leader_election.cc:288] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:01.522238 25326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:01.522529 25326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [term 1 LEADER]: Becoming Leader. State: Replica: 20f7dbcfaac64b28900bd39440942f9c, State: Running, Role: LEADER
I20260810 22:33:01.522886 25326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f7dbcfaac64b28900bd39440942f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42981 } }
I20260810 22:33:01.523523 25314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:01.524664 25328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20f7dbcfaac64b28900bd39440942f9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f7dbcfaac64b28900bd39440942f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42981 } } }
I20260810 22:33:01.524637 25327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20f7dbcfaac64b28900bd39440942f9c. Latest consensus state: current_term: 1 leader_uuid: "20f7dbcfaac64b28900bd39440942f9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f7dbcfaac64b28900bd39440942f9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42981 } } }
I20260810 22:33:01.524773 25327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:01.524773 25328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:01.525183 25337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:01.528379 25337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:01.533829 25337 catalog_manager.cc:1347] Generated new cluster ID: 9f06b1a1b223449588920372c4dad1c7
I20260810 22:33:01.533905 25337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:01.545643 25337 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:01.545951 25337 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:01.551795 25337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20f7dbcfaac64b28900bd39440942f9c: Generated new TSK 0
I20260810 22:33:01.552507 25337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:01.562739 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 25237
2026-08-10T22:33:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:01 dist-test-slave-njxd krb5kdc[25349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:01 dist-test-slave-njxd krb5kdc[25349](info): set up 2 sockets
Aug 10 22:33:01 dist-test-slave-njxd krb5kdc[25349](info): commencing operation
krb5kdc: starting...
W20260810 22:33:02.665934 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:02 dist-test-slave-njxd krb5kdc[25349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:33:02.796739 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:02.909327 25362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:02.909682 25362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:02.910039 25362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:02.914160 25362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:02.914249 25362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:02.914296 25362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:02.914353 25362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:02.914391 25362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:02.919651 25362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:02.920785 25362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:02.921924 25362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:02.930795 25368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:02.930802 25369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:02.930816 25371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:02.930863 25362 server_base.cc:1034] running on GCE node
I20260810 22:33:02.931635 25362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:02.932772 25362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:02.933938 25362 hybrid_clock.cc:648] HybridClock initialized: now 1786401182933909 us; error 42 us; skew 500 ppm
Aug 10 22:33:02 dist-test-slave-njxd krb5kdc[25349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:02.941825 25362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:33:02.942852 25362 webserver.cc:460] Webserver started at http://127.0.0.1:38657/ using document root <none> and password file <none>
I20260810 22:33:02.943419 25362 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:02.943483 25362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:02.943661 25362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:02.945879 25362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "7ddbfdcdb0d94f1d83d501602c65e930"
format_stamp: "Formatted at 2026-08-10 22:33:02 on dist-test-slave-njxd"
server_key: "da3163d2fcfd59d1d03507f4f0dc1c1f"
server_key_iv: "4f2c1495a42ea88d801c8cf8029117c0"
server_key_version: "encryptionkey@0"
I20260810 22:33:02.946439 25362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "7ddbfdcdb0d94f1d83d501602c65e930"
format_stamp: "Formatted at 2026-08-10 22:33:02 on dist-test-slave-njxd"
server_key: "da3163d2fcfd59d1d03507f4f0dc1c1f"
server_key_iv: "4f2c1495a42ea88d801c8cf8029117c0"
server_key_version: "encryptionkey@0"
I20260810 22:33:02.950259 25362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:33:02.952679 25379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:02.954020 25362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:33:02.954166 25362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "7ddbfdcdb0d94f1d83d501602c65e930"
format_stamp: "Formatted at 2026-08-10 22:33:02 on dist-test-slave-njxd"
server_key: "da3163d2fcfd59d1d03507f4f0dc1c1f"
server_key_iv: "4f2c1495a42ea88d801c8cf8029117c0"
server_key_version: "encryptionkey@0"
I20260810 22:33:02.954296 25362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:02.965936 25362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:02.968796 25362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:02.969005 25362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:02.976787 25362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43943
I20260810 22:33:02.976815 25434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43943 every 8 connection(s)
I20260810 22:33:02.977793 25362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:02.980638 25435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:02.984659 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 25362
I20260810 22:33:02.984810 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:02.985101 1910 external_mini_cluster.cc:1428] Setting key f01b49f8d6d773fbfa1f2ddedaf63635
I20260810 22:33:02.987095 25435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930: Bootstrap starting.
I20260810 22:33:02.989989 25435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:02.990916 25435 log.cc:824] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:02.993019 25435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930: No bootstrap required, opened a new log
Aug 10 22:33:02 dist-test-slave-njxd krb5kdc[25349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 22:33:02.997633 25435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ddbfdcdb0d94f1d83d501602c65e930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260810 22:33:02.997923 25435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:02.998025 25435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ddbfdcdb0d94f1d83d501602c65e930, State: Initialized, Role: FOLLOWER
I20260810 22:33:02.998499 25435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ddbfdcdb0d94f1d83d501602c65e930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260810 22:33:02.998659 25435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:02.998737 25435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:02.998871 25435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:02.999810 25435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ddbfdcdb0d94f1d83d501602c65e930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260810 22:33:03.000249 25435 leader_election.cc:302] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ddbfdcdb0d94f1d83d501602c65e930; no voters:
I20260810 22:33:03.000645 25435 leader_election.cc:288] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:03.000751 25440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:03.000969 25440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [term 1 LEADER]: Becoming Leader. State: Replica: 7ddbfdcdb0d94f1d83d501602c65e930, State: Running, Role: LEADER
I20260810 22:33:03.001169 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:02.986869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:37383 (local address 127.0.0.1:43943)
0810 22:33:02.987339 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:02.987361 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:02.987407 (+ 46us) server_negotiation.cc:407] Connection header received
0810 22:33:02.988277 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:02.988313 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:02.988781 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:02.989140 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:02.989941 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:02.991455 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:02.992388 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:02.992874 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:02.993049 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:02.995987 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:02.995992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:02.996006 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:02.996031 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:02.997860 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:02.998457 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:02.998470 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:02.998475 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:02.998587 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:02.998933 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:02.998943 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:02.998945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:02.999372 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:02.999620 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:02.999938 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 22:33:03.000167 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":136,"threads_started":1}
I20260810 22:33:03.001489 25440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ddbfdcdb0d94f1d83d501602c65e930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } }
I20260810 22:33:03.002012 25435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:03.003229 25442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ddbfdcdb0d94f1d83d501602c65e930. Latest consensus state: current_term: 1 leader_uuid: "7ddbfdcdb0d94f1d83d501602c65e930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ddbfdcdb0d94f1d83d501602c65e930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } } }
I20260810 22:33:03.003362 25442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:03.003727 25441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ddbfdcdb0d94f1d83d501602c65e930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ddbfdcdb0d94f1d83d501602c65e930" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43943 } } }
I20260810 22:33:03.003821 25441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:03.003878 25449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:03.004650 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:03.007071 25449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:03.012873 25449 catalog_manager.cc:1347] Generated new cluster ID: 7fda4cd485fa4946a0ffa7e1eeca0178
I20260810 22:33:03.012929 25449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:03.016120 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:03.007452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:45469 (local address 127.0.0.1:43943)
0810 22:33:03.007576 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:03.007580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:03.007892 (+ 312us) server_negotiation.cc:407] Connection header received
0810 22:33:03.008038 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:03.008041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:03.008091 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:03.008178 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:03.008837 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:03.009627 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:03.010824 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:03.011160 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:03.011276 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:03.012754 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:03.012757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:03.012760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:03.012788 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:03.014285 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:03.014797 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:03.014803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:03.014804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:03.014883 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:03.015226 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:03.015230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:03.015231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:03.015399 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:03.015552 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:03.015829 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 22:33:03.015932 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 22:33:03.034379 25449 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:03.035354 25449 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:03.041896 25449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ddbfdcdb0d94f1d83d501602c65e930: Generated new TSK 0
I20260810 22:33:03.042598 25449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:03.050841 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 25362
W20260810 22:33:03.059289 25347 connection.cc:535] client connection to 127.0.0.1:43943 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43943): Connection reset by peer (error 104)
2026-08-10T22:33:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1487 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:03 dist-test-slave-njxd krb5kdc[25470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:03 dist-test-slave-njxd krb5kdc[25470](info): set up 2 sockets
Aug 10 22:33:03 dist-test-slave-njxd krb5kdc[25470](info): commencing operation
krb5kdc: starting...
W20260810 22:33:04.146063 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:04 dist-test-slave-njxd krb5kdc[25470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:33:04.280857 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:04.396425 25483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:04.396718 25483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:04.396799 25483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:04.400952 25483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:04.401013 25483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:04.401034 25483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:04.401099 25483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:04.401125 25483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:04.406596 25483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:04.407706 25483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:04.408819 25483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:04.417562 25491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:04.417543 25489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:04.417769 25483 server_base.cc:1034] running on GCE node
W20260810 22:33:04.417547 25488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:04.418504 25483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:04.419721 25483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:04.420914 25483 hybrid_clock.cc:648] HybridClock initialized: now 1786401184420883 us; error 45 us; skew 500 ppm
Aug 10 22:33:04 dist-test-slave-njxd krb5kdc[25470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:04.423867 25483 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:33:04.424857 25483 webserver.cc:460] Webserver started at http://127.0.0.1:39345/ using document root <none> and password file <none>
I20260810 22:33:04.425441 25483 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:04.425534 25483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:04.425812 25483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:04.427937 25483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "29d1adf85bde4309a63c4c7ca354c204"
format_stamp: "Formatted at 2026-08-10 22:33:04 on dist-test-slave-njxd"
server_key: "bc1760c89b5618bf50e4ae0c09a61b07"
server_key_iv: "d8996965277d51db2b2857d2b0e44cde"
server_key_version: "encryptionkey@0"
I20260810 22:33:04.428586 25483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "29d1adf85bde4309a63c4c7ca354c204"
format_stamp: "Formatted at 2026-08-10 22:33:04 on dist-test-slave-njxd"
server_key: "bc1760c89b5618bf50e4ae0c09a61b07"
server_key_iv: "d8996965277d51db2b2857d2b0e44cde"
server_key_version: "encryptionkey@0"
I20260810 22:33:04.432590 25483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 22:33:04.435369 25498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:04.436638 25483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 22:33:04.436797 25483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "29d1adf85bde4309a63c4c7ca354c204"
format_stamp: "Formatted at 2026-08-10 22:33:04 on dist-test-slave-njxd"
server_key: "bc1760c89b5618bf50e4ae0c09a61b07"
server_key_iv: "d8996965277d51db2b2857d2b0e44cde"
server_key_version: "encryptionkey@0"
I20260810 22:33:04.436924 25483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:04.456188 25483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:04.459837 25483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:04.460059 25483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:04.467828 25483 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41143
I20260810 22:33:04.467856 25554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41143 every 8 connection(s)
I20260810 22:33:04.468861 25483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:04.471689 25555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:04.477488 25555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204: Bootstrap starting.
I20260810 22:33:04.478359 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 25483
I20260810 22:33:04.478497 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:04.478776 1910 external_mini_cluster.cc:1428] Setting key 963d4ae2b17c32957ace8426238c312d
I20260810 22:33:04.480377 25555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:04.481129 25555 log.cc:824] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:04.483223 25555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204: No bootstrap required, opened a new log
I20260810 22:33:04.486861 25555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d1adf85bde4309a63c4c7ca354c204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260810 22:33:04.487115 25555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:04.487198 25555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d1adf85bde4309a63c4c7ca354c204, State: Initialized, Role: FOLLOWER
I20260810 22:33:04.487581 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:04.479892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:47093 (local address 127.0.0.1:41143)
0810 22:33:04.480350 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:04.480366 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:04.480400 (+ 34us) server_negotiation.cc:407] Connection header received
0810 22:33:04.481087 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:04.481105 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:04.481537 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:04.481987 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:04.482315 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:04.483251 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:04.484148 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:04.484618 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:04.484820 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:04.485778 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:04.485783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:04.485796 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:04.486173 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:04.486226 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:04.486248 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:04.486391 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:04.486578 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 22:33:04.486726 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":153,"threads_started":1}
I20260810 22:33:04.487738 25555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d1adf85bde4309a63c4c7ca354c204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260810 22:33:04.487907 25555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:04.487980 25555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:04.488077 25555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:04.489099 25555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d1adf85bde4309a63c4c7ca354c204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260810 22:33:04.489464 25555 leader_election.cc:302] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29d1adf85bde4309a63c4c7ca354c204; no voters:
I20260810 22:33:04.489886 25555 leader_election.cc:288] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:04.489962 25561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:04.490182 25561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [term 1 LEADER]: Becoming Leader. State: Replica: 29d1adf85bde4309a63c4c7ca354c204, State: Running, Role: LEADER
I20260810 22:33:04.490513 25561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d1adf85bde4309a63c4c7ca354c204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260810 22:33:04.490638 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:04.491230 25555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:04.493016 25563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29d1adf85bde4309a63c4c7ca354c204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d1adf85bde4309a63c4c7ca354c204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } } }
I20260810 22:33:04.493284 25563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:04.493719 25562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29d1adf85bde4309a63c4c7ca354c204. Latest consensus state: current_term: 1 leader_uuid: "29d1adf85bde4309a63c4c7ca354c204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d1adf85bde4309a63c4c7ca354c204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } } }
I20260810 22:33:04.493757 25582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:04.493870 25562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:04.497251 25582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:04.498353 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:04.493736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:45021 (local address 127.0.0.1:41143)
0810 22:33:04.493870 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:04.493876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:04.494588 (+ 712us) server_negotiation.cc:407] Connection header received
0810 22:33:04.494704 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:04.494706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:04.494749 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:04.494820 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:04.495134 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:04.495650 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:04.496503 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:04.496750 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:04.496870 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:04.497885 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:04.497888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:04.497890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:04.497903 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:04.497929 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:04.497940 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:04.498041 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:04.498173 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 22:33:04.498205 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:33:04.503283 25582 catalog_manager.cc:1347] Generated new cluster ID: f727b3ca155d4f62944ee187d9f01a51
I20260810 22:33:04.503361 25582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:04.513917 25582 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:04.515117 25582 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:04.520292 25582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29d1adf85bde4309a63c4c7ca354c204: Generated new TSK 0
I20260810 22:33:04.520999 25582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:04.530758 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 25483
W20260810 22:33:04.539837 25468 connection.cc:535] client connection to 127.0.0.1:41143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41143): Connection reset by peer (error 104)
2026-08-10T22:33:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1480 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:04 dist-test-slave-njxd krb5kdc[25589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:04 dist-test-slave-njxd krb5kdc[25589](info): set up 2 sockets
Aug 10 22:33:04 dist-test-slave-njxd krb5kdc[25589](info): commencing operation
krb5kdc: starting...
W20260810 22:33:05.631657 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:05 dist-test-slave-njxd krb5kdc[25589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:33:05.760210 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:05.874361 25603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:05.874617 25603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:05.874660 25603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:05.878669 25603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:05.878721 25603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:05.878741 25603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:05.878756 25603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:05.878801 25603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:05.883836 25603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:05.884860 25603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:05.885941 25603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:05.894235 25610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:05.894253 25614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:05.894241 25612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:05.894413 25603 server_base.cc:1034] running on GCE node
I20260810 22:33:05.895107 25603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:05.896179 25603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:05.897338 25603 hybrid_clock.cc:648] HybridClock initialized: now 1786401185897319 us; error 30 us; skew 500 ppm
Aug 10 22:33:05 dist-test-slave-njxd krb5kdc[25589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:05.900045 25603 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:33:05.900908 25603 webserver.cc:460] Webserver started at http://127.0.0.1:39459/ using document root <none> and password file <none>
I20260810 22:33:05.901408 25603 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:05.901479 25603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:05.901710 25603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:05.903738 25603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "5e3480b97c2049a4a7e14348ae1d5240"
format_stamp: "Formatted at 2026-08-10 22:33:05 on dist-test-slave-njxd"
server_key: "18b3894009e2b3ef51a48e01e0ee338f"
server_key_iv: "db5cf35064db272e3a0de5019f2c2bc8"
server_key_version: "encryptionkey@0"
I20260810 22:33:05.904265 25603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "5e3480b97c2049a4a7e14348ae1d5240"
format_stamp: "Formatted at 2026-08-10 22:33:05 on dist-test-slave-njxd"
server_key: "18b3894009e2b3ef51a48e01e0ee338f"
server_key_iv: "db5cf35064db272e3a0de5019f2c2bc8"
server_key_version: "encryptionkey@0"
I20260810 22:33:05.908077 25603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:33:05.910440 25621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:05.911582 25603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:33:05.911689 25603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "5e3480b97c2049a4a7e14348ae1d5240"
format_stamp: "Formatted at 2026-08-10 22:33:05 on dist-test-slave-njxd"
server_key: "18b3894009e2b3ef51a48e01e0ee338f"
server_key_iv: "db5cf35064db272e3a0de5019f2c2bc8"
server_key_version: "encryptionkey@0"
I20260810 22:33:05.911772 25603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:05.916088 25603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:05.918835 25603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:05.918972 25603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:05.925951 25603 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46455
I20260810 22:33:05.925973 25679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46455 every 8 connection(s)
I20260810 22:33:05.926898 25603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:05.929527 25680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:05.935082 25680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240: Bootstrap starting.
I20260810 22:33:05.936659 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 25603
I20260810 22:33:05.936810 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:05.937108 1910 external_mini_cluster.cc:1428] Setting key 3299a36a23c899c57b8ea42bcac419a5
I20260810 22:33:05.937691 25680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:05.938520 25680 log.cc:824] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:05.940510 25680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240: No bootstrap required, opened a new log
I20260810 22:33:05.942315 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:05.938283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:44621 (local address 127.0.0.1:46455)
0810 22:33:05.938750 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:05.938773 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:05.938807 (+ 34us) server_negotiation.cc:407] Connection header received
0810 22:33:05.939591 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:05.939610 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:05.939887 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:05.940154 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:05.941038 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:05.941043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:05.941054 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:05.941064 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:05.941132 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:05.941152 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:05.941237 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:05.941386 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 22:33:05.941575 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":164,"threads_started":1}
I20260810 22:33:05.944729 25680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3480b97c2049a4a7e14348ae1d5240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46455 } }
I20260810 22:33:05.944990 25680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:05.945109 25680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e3480b97c2049a4a7e14348ae1d5240, State: Initialized, Role: FOLLOWER
I20260810 22:33:05.945255 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:05.945636 25680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3480b97c2049a4a7e14348ae1d5240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46455 } }
I20260810 22:33:05.945801 25680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:05.945869 25680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:05.945973 25680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:05.947335 25680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3480b97c2049a4a7e14348ae1d5240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46455 } }
I20260810 22:33:05.947676 25680 leader_election.cc:302] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e3480b97c2049a4a7e14348ae1d5240; no voters:
I20260810 22:33:05.948014 25680 leader_election.cc:288] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:05.948360 25692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:05.948626 25692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [term 1 LEADER]: Becoming Leader. State: Replica: 5e3480b97c2049a4a7e14348ae1d5240, State: Running, Role: LEADER
I20260810 22:33:05.948913 25692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3480b97c2049a4a7e14348ae1d5240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46455 } }
I20260810 22:33:05.949509 25680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:05.949733 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:05.947823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:53719 (local address 127.0.0.1:46455)
0810 22:33:05.947943 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:05.947950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:05.948119 (+ 169us) server_negotiation.cc:407] Connection header received
0810 22:33:05.948291 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:05.948294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:05.948325 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:05.948415 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:05.949273 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:05.949276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:05.949279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:05.949290 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:05.949315 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:05.949326 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:05.949396 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:05.949543 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 22:33:05.949605 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:33:05.950392 25692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e3480b97c2049a4a7e14348ae1d5240. Latest consensus state: current_term: 1 leader_uuid: "5e3480b97c2049a4a7e14348ae1d5240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3480b97c2049a4a7e14348ae1d5240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46455 } } }
I20260810 22:33:05.950585 25692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:05.950789 25693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e3480b97c2049a4a7e14348ae1d5240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3480b97c2049a4a7e14348ae1d5240" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46455 } } }
I20260810 22:33:05.951028 25693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:05.951383 25700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:05.954270 25700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:05.960026 25700 catalog_manager.cc:1347] Generated new cluster ID: fdad7189c2ec42958f6dd470c022eede
I20260810 22:33:05.960103 25700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:05.986697 25700 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:05.987004 25700 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:05.996213 25700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e3480b97c2049a4a7e14348ae1d5240: Generated new TSK 0
I20260810 22:33:05.996878 25700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:06.000633 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 25603
2026-08-10T22:33:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:06 dist-test-slave-njxd krb5kdc[25713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:06 dist-test-slave-njxd krb5kdc[25713](info): set up 2 sockets
Aug 10 22:33:06 dist-test-slave-njxd krb5kdc[25713](info): commencing operation
krb5kdc: starting...
W20260810 22:33:07.096460 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:07 dist-test-slave-njxd krb5kdc[25713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:33:07.224954 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:07.340790 25726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:07.341023 25726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:07.341063 25726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:07.344774 25726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:07.344827 25726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:07.344846 25726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:07.344861 25726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:07.344877 25726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:07.350009 25726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:07.350962 25726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:07.351933 25726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:07.360232 25734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:07.360311 25726 server_base.cc:1034] running on GCE node
W20260810 22:33:07.360232 25735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:07.360226 25737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:07.361111 25726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:07.362326 25726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:07.363511 25726 hybrid_clock.cc:648] HybridClock initialized: now 1786401187363481 us; error 50 us; skew 500 ppm
Aug 10 22:33:07 dist-test-slave-njxd krb5kdc[25713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:07.366212 25726 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:33:07.367080 25726 webserver.cc:460] Webserver started at http://127.0.0.1:44433/ using document root <none> and password file <none>
I20260810 22:33:07.367694 25726 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:07.367785 25726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:07.368021 25726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:07.370148 25726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "d322e70146274e089fe7c2bb39fa6d49"
format_stamp: "Formatted at 2026-08-10 22:33:07 on dist-test-slave-njxd"
server_key: "95e3ac01e793c6646e98b91cbfebb726"
server_key_iv: "fbed01ec4d6c1ca0978b42523db89efb"
server_key_version: "encryptionkey@0"
I20260810 22:33:07.370723 25726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "d322e70146274e089fe7c2bb39fa6d49"
format_stamp: "Formatted at 2026-08-10 22:33:07 on dist-test-slave-njxd"
server_key: "95e3ac01e793c6646e98b91cbfebb726"
server_key_iv: "fbed01ec4d6c1ca0978b42523db89efb"
server_key_version: "encryptionkey@0"
I20260810 22:33:07.374451 25726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:33:07.376904 25744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:07.378248 25726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:33:07.378389 25726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "d322e70146274e089fe7c2bb39fa6d49"
format_stamp: "Formatted at 2026-08-10 22:33:07 on dist-test-slave-njxd"
server_key: "95e3ac01e793c6646e98b91cbfebb726"
server_key_iv: "fbed01ec4d6c1ca0978b42523db89efb"
server_key_version: "encryptionkey@0"
I20260810 22:33:07.378507 25726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:07.397418 25726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:07.400430 25726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:07.400653 25726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:07.408061 25726 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42315
I20260810 22:33:07.408113 25799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42315 every 8 connection(s)
I20260810 22:33:07.409067 25726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:07.411828 25800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:07.412462 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 25726
I20260810 22:33:07.412629 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:07.412921 1910 external_mini_cluster.cc:1428] Setting key bfc9862bcdb9ec4e44b2933695c19d0c
I20260810 22:33:07.418062 25800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49: Bootstrap starting.
I20260810 22:33:07.420784 25800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:07.421752 25800 log.cc:824] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:33:07 dist-test-slave-njxd krb5kdc[25713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 22:33:07.423842 25800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49: No bootstrap required, opened a new log
I20260810 22:33:07.427467 25800 raft_consensus.cc:348] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d322e70146274e089fe7c2bb39fa6d49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42315 } }
I20260810 22:33:07.427726 25800 raft_consensus.cc:374] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:07.427834 25800 raft_consensus.cc:729] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d322e70146274e089fe7c2bb39fa6d49, State: Initialized, Role: FOLLOWER
I20260810 22:33:07.428079 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:07.414387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:42475 (local address 127.0.0.1:42315)
0810 22:33:07.414873 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:07.414904 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:07.414940 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:33:07.415577 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:07.415593 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:07.415934 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:07.416198 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:07.417080 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:07.418137 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:07.419357 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:07.419844 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:07.419982 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:07.423086 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:07.423091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:07.423104 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:07.423130 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:07.424929 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:07.425477 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:07.425482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:07.425487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:07.425670 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:07.426031 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:07.426034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:07.426036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:07.426415 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:07.426673 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:07.426952 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 22:33:07.427172 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":208,"threads_started":1}
I20260810 22:33:07.428328 25800 consensus_queue.cc:260] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d322e70146274e089fe7c2bb39fa6d49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42315 } }
I20260810 22:33:07.428478 25800 raft_consensus.cc:388] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:07.428565 25800 raft_consensus.cc:482] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:07.428670 25800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:07.429790 25800 raft_consensus.cc:504] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d322e70146274e089fe7c2bb39fa6d49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42315 } }
I20260810 22:33:07.430142 25800 leader_election.cc:302] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d322e70146274e089fe7c2bb39fa6d49; no voters:
I20260810 22:33:07.430474 25800 leader_election.cc:288] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:07.430600 25806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:07.430827 25806 raft_consensus.cc:686] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [term 1 LEADER]: Becoming Leader. State: Replica: d322e70146274e089fe7c2bb39fa6d49, State: Running, Role: LEADER
I20260810 22:33:07.431252 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:07.431201 25806 consensus_queue.cc:237] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d322e70146274e089fe7c2bb39fa6d49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42315 } }
I20260810 22:33:07.431761 25800 sys_catalog.cc:564] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:07.432912 25807 sys_catalog.cc:455] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d322e70146274e089fe7c2bb39fa6d49. Latest consensus state: current_term: 1 leader_uuid: "d322e70146274e089fe7c2bb39fa6d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d322e70146274e089fe7c2bb39fa6d49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42315 } } }
I20260810 22:33:07.433033 25807 sys_catalog.cc:458] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:07.433506 25822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:07.433487 25808 sys_catalog.cc:455] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d322e70146274e089fe7c2bb39fa6d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d322e70146274e089fe7c2bb39fa6d49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42315 } } }
I20260810 22:33:07.433704 25808 sys_catalog.cc:458] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:07.436992 25822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:07.443552 25822 catalog_manager.cc:1347] Generated new cluster ID: 33a3c11b98e84b7abf24356cc80e6504
I20260810 22:33:07.443619 25822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:07.444409 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:07.434248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:47577 (local address 127.0.0.1:42315)
0810 22:33:07.434368 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:07.434373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:07.436023 (+ 1650us) server_negotiation.cc:407] Connection header received
0810 22:33:07.436216 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:07.436222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:07.436276 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:07.436350 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:07.437053 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:07.437653 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:07.438530 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:07.438850 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:07.438983 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:07.440484 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:07.440488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:07.440490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:07.440515 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:07.442283 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:07.442789 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:07.442792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:07.442794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:07.442886 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:07.443269 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:07.443275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:07.443279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:07.443523 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:07.443680 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:07.444115 (+ 435us) server_negotiation.cc:299] Negotiation successful
0810 22:33:07.444232 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:33:07.455487 25822 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:07.456409 25822 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:07.463510 25822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d322e70146274e089fe7c2bb39fa6d49: Generated new TSK 0
I20260810 22:33:07.464213 25822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:07.475171 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 25726
W20260810 22:33:07.484336 25711 connection.cc:535] client connection to 127.0.0.1:42315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42315): Connection reset by peer (error 104)
2026-08-10T22:33:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:07 dist-test-slave-njxd krb5kdc[25835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:07 dist-test-slave-njxd krb5kdc[25835](info): set up 2 sockets
Aug 10 22:33:07 dist-test-slave-njxd krb5kdc[25835](info): commencing operation
krb5kdc: starting...
W20260810 22:33:08.572402 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:08 dist-test-slave-njxd krb5kdc[25835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:33:08.702286 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:08.815593 25848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:08.815927 25848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:08.816002 25848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:08.820472 25848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:08.820577 25848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:08.820618 25848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:08.820655 25848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:08.820691 25848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:08.826270 25848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:08.827543 25848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:08.828667 25848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:08.837450 25855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:08.837626 25857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:08.837450 25854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:08.837527 25848 server_base.cc:1034] running on GCE node
I20260810 22:33:08.838349 25848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:08.839488 25848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:08.840652 25848 hybrid_clock.cc:648] HybridClock initialized: now 1786401188840633 us; error 33 us; skew 500 ppm
Aug 10 22:33:08 dist-test-slave-njxd krb5kdc[25835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:08.850796 25848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:33:08.851927 25848 webserver.cc:460] Webserver started at http://127.0.0.1:41709/ using document root <none> and password file <none>
I20260810 22:33:08.852507 25848 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:08.852569 25848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:08.852747 25848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:08.854955 25848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "71734a44d2b242ce886f25a39df1d11b"
format_stamp: "Formatted at 2026-08-10 22:33:08 on dist-test-slave-njxd"
server_key: "35cf89ffc3b008952ed61385833d9cba"
server_key_iv: "a41907249733ee8b9dada785ad0cb521"
server_key_version: "encryptionkey@0"
I20260810 22:33:08.855496 25848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "71734a44d2b242ce886f25a39df1d11b"
format_stamp: "Formatted at 2026-08-10 22:33:08 on dist-test-slave-njxd"
server_key: "35cf89ffc3b008952ed61385833d9cba"
server_key_iv: "a41907249733ee8b9dada785ad0cb521"
server_key_version: "encryptionkey@0"
I20260810 22:33:08.859373 25848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 22:33:08.861797 25864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:08.863080 25848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:33:08.863212 25848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "71734a44d2b242ce886f25a39df1d11b"
format_stamp: "Formatted at 2026-08-10 22:33:08 on dist-test-slave-njxd"
server_key: "35cf89ffc3b008952ed61385833d9cba"
server_key_iv: "a41907249733ee8b9dada785ad0cb521"
server_key_version: "encryptionkey@0"
I20260810 22:33:08.863317 25848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:08.881062 25848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:08.883985 25848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:08.884209 25848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:08.891619 25848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39583
I20260810 22:33:08.891635 25918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39583 every 8 connection(s)
I20260810 22:33:08.892567 25848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:08.895361 25919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:08.899119 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 25848
I20260810 22:33:08.899271 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:08.899581 1910 external_mini_cluster.cc:1428] Setting key 1fe5a3d5e99a22bf04fc39afa917b690
I20260810 22:33:08.901233 25919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b: Bootstrap starting.
I20260810 22:33:08.903607 25919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:08.904461 25919 log.cc:824] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:08.906713 25919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b: No bootstrap required, opened a new log
I20260810 22:33:08.908684 25923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:08.900913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:38931 (local address 127.0.0.1:39583)
0810 22:33:08.901311 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:08.901327 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:08.901385 (+ 58us) server_negotiation.cc:407] Connection header received
0810 22:33:08.902070 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:08.902089 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:08.902437 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:08.902714 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:08.903041 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:08.904073 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:08.905239 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:08.905771 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:08.905889 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:08.906785 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:08.906790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:08.906820 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:08.907219 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:08.907274 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:08.907296 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:08.907400 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:08.907606 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 22:33:08.907763 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":121,"threads_started":1}
I20260810 22:33:08.909896 25919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71734a44d2b242ce886f25a39df1d11b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39583 } }
I20260810 22:33:08.910161 25919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:08.910216 25919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71734a44d2b242ce886f25a39df1d11b, State: Initialized, Role: FOLLOWER
I20260810 22:33:08.910620 25919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71734a44d2b242ce886f25a39df1d11b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39583 } }
I20260810 22:33:08.910794 25919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:08.910868 25919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:08.911005 25919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:08.911677 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:08.912142 25919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71734a44d2b242ce886f25a39df1d11b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39583 } }
I20260810 22:33:08.912546 25919 leader_election.cc:302] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71734a44d2b242ce886f25a39df1d11b; no voters:
I20260810 22:33:08.912902 25919 leader_election.cc:288] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:08.913024 25925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:08.913287 25925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [term 1 LEADER]: Becoming Leader. State: Replica: 71734a44d2b242ce886f25a39df1d11b, State: Running, Role: LEADER
I20260810 22:33:08.913697 25925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71734a44d2b242ce886f25a39df1d11b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39583 } }
I20260810 22:33:08.914239 25919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:08.915524 25928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71734a44d2b242ce886f25a39df1d11b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71734a44d2b242ce886f25a39df1d11b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39583 } } }
I20260810 22:33:08.915649 25928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:08.915886 25928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71734a44d2b242ce886f25a39df1d11b. Latest consensus state: current_term: 1 leader_uuid: "71734a44d2b242ce886f25a39df1d11b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71734a44d2b242ce886f25a39df1d11b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39583 } } }
I20260810 22:33:08.915959 25928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:08.919072 25923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:08.913977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:38811 (local address 127.0.0.1:39583)
0810 22:33:08.914074 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:08.914078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:08.914503 (+ 425us) server_negotiation.cc:407] Connection header received
0810 22:33:08.914645 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:08.914648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:08.914705 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:08.914790 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:08.915119 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:08.915968 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:08.917287 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:08.917518 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:08.917649 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:08.918518 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:08.918521 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:08.918524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:08.918536 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:08.918562 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:08.918575 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:08.918633 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:08.918908 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 22:33:08.918945 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260810 22:33:08.919356 25948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:33:08.919436 25948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:33:08.920009 25940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:08.920845 25940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:08.925560 25940 catalog_manager.cc:1347] Generated new cluster ID: b7f5d87a8d6a4db3a9b8b811252cbd3b
I20260810 22:33:08.925645 25940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:08.936959 25940 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:08.938289 25940 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:08.943298 25940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71734a44d2b242ce886f25a39df1d11b: Generated new TSK 0
I20260810 22:33:08.943986 25940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:08.953758 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 25848
W20260810 22:33:08.961503 25833 connection.cc:535] client connection to 127.0.0.1:39583 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39583): Connection reset by peer (error 104)
2026-08-10T22:33:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:09 dist-test-slave-njxd krb5kdc[25955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:09 dist-test-slave-njxd krb5kdc[25955](info): set up 2 sockets
Aug 10 22:33:09 dist-test-slave-njxd krb5kdc[25955](info): commencing operation
krb5kdc: starting...
W20260810 22:33:10.040285 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:10 dist-test-slave-njxd krb5kdc[25955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 22:33:10.162407 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:10.271971 25969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:10.272294 25969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:10.272372 25969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:10.276549 25969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:10.276644 25969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:10.276681 25969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:10.276732 25969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:10.276769 25969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:10.282191 25969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:10.283351 25969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:10.284446 25969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:10.293181 25978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:10.293186 25975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:10.293181 25976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:10.293455 25969 server_base.cc:1034] running on GCE node
I20260810 22:33:10.294152 25969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:10.295290 25969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:10.296491 25969 hybrid_clock.cc:648] HybridClock initialized: now 1786401190296478 us; error 32 us; skew 500 ppm
Aug 10 22:33:10 dist-test-slave-njxd krb5kdc[25955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:10.299386 25969 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 22:33:10.300402 25969 webserver.cc:460] Webserver started at http://127.0.0.1:44315/ using document root <none> and password file <none>
I20260810 22:33:10.301008 25969 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:10.301105 25969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:10.301347 25969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:10.303530 25969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "e22994cb77824db2903ea84ecab2c584"
format_stamp: "Formatted at 2026-08-10 22:33:10 on dist-test-slave-njxd"
server_key: "8a67afe80b4704dfb4c6db431ba86e82"
server_key_iv: "a2b7004b9e6c8de90f74f80a6692035b"
server_key_version: "encryptionkey@0"
I20260810 22:33:10.304122 25969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "e22994cb77824db2903ea84ecab2c584"
format_stamp: "Formatted at 2026-08-10 22:33:10 on dist-test-slave-njxd"
server_key: "8a67afe80b4704dfb4c6db431ba86e82"
server_key_iv: "a2b7004b9e6c8de90f74f80a6692035b"
server_key_version: "encryptionkey@0"
I20260810 22:33:10.308053 25969 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 22:33:10.310580 25985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:10.311934 25969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:33:10.312108 25969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "e22994cb77824db2903ea84ecab2c584"
format_stamp: "Formatted at 2026-08-10 22:33:10 on dist-test-slave-njxd"
server_key: "8a67afe80b4704dfb4c6db431ba86e82"
server_key_iv: "a2b7004b9e6c8de90f74f80a6692035b"
server_key_version: "encryptionkey@0"
I20260810 22:33:10.312228 25969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:10.317082 25969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:10.320453 25969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:10.320650 25969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:10.328066 25969 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46077
I20260810 22:33:10.328095 26043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46077 every 8 connection(s)
I20260810 22:33:10.329026 25969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:10.331861 26044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:10.337646 26044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584: Bootstrap starting.
I20260810 22:33:10.338440 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 25969
I20260810 22:33:10.338588 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:10.338932 1910 external_mini_cluster.cc:1428] Setting key a04d85c2216d2ef59eecf169318244a8
I20260810 22:33:10.340808 26044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:10.341560 26044 log.cc:824] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:10.344007 26044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584: No bootstrap required, opened a new log
I20260810 22:33:10.344503 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:10.340252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:55833 (local address 127.0.0.1:46077)
0810 22:33:10.340717 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:10.340732 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:10.340770 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:33:10.341490 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:10.341509 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:10.341953 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:10.342370 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:10.343176 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:10.343180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:10.343192 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:10.343207 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:10.343256 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:10.343294 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:10.343387 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:10.343541 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 22:33:10.343724 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":133,"threads_started":1}
I20260810 22:33:10.347570 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:10.348383 26044 raft_consensus.cc:348] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22994cb77824db2903ea84ecab2c584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } }
I20260810 22:33:10.348635 26044 raft_consensus.cc:374] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:10.348706 26044 raft_consensus.cc:729] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22994cb77824db2903ea84ecab2c584, State: Initialized, Role: FOLLOWER
I20260810 22:33:10.349220 26044 consensus_queue.cc:260] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22994cb77824db2903ea84ecab2c584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } }
I20260810 22:33:10.349442 26044 raft_consensus.cc:388] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:10.349524 26044 raft_consensus.cc:482] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:10.349656 26044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:10.350973 26044 raft_consensus.cc:504] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22994cb77824db2903ea84ecab2c584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } }
I20260810 22:33:10.351341 26044 leader_election.cc:302] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e22994cb77824db2903ea84ecab2c584; no voters:
I20260810 22:33:10.351601 26044 leader_election.cc:288] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:10.351771 26056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:10.352008 26056 raft_consensus.cc:686] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [term 1 LEADER]: Becoming Leader. State: Replica: e22994cb77824db2903ea84ecab2c584, State: Running, Role: LEADER
I20260810 22:33:10.352331 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:10.350321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.106:53633 (local address 127.0.0.1:46077)
0810 22:33:10.350481 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:10.350486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:10.350826 (+ 340us) server_negotiation.cc:407] Connection header received
0810 22:33:10.351018 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:10.351022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:10.351065 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:10.351153 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:10.351880 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:10.351884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:10.351887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:10.351900 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:10.351930 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:10.351942 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:10.352026 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:10.352177 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 22:33:10.352221 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:33:10.352344 26056 consensus_queue.cc:237] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22994cb77824db2903ea84ecab2c584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } }
I20260810 22:33:10.352938 26044 sys_catalog.cc:564] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:10.353547 26056 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e22994cb77824db2903ea84ecab2c584. Latest consensus state: current_term: 1 leader_uuid: "e22994cb77824db2903ea84ecab2c584" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22994cb77824db2903ea84ecab2c584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } } }
I20260810 22:33:10.353693 26056 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:10.354135 26058 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e22994cb77824db2903ea84ecab2c584" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22994cb77824db2903ea84ecab2c584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } } }
I20260810 22:33:10.354241 26058 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:10.355510 26063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:10.358188 26063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:10.363299 26063 catalog_manager.cc:1347] Generated new cluster ID: 99b6a82fb96e439e8c05fb0d7f05c85b
I20260810 22:33:10.363370 26063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:10.379419 26063 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:10.379765 26063 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:10.394191 26063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e22994cb77824db2903ea84ecab2c584: Generated new TSK 0
I20260810 22:33:10.394908 26063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:10.401101 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 25969
2026-08-10T22:33:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:10 dist-test-slave-njxd krb5kdc[26080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:10 dist-test-slave-njxd krb5kdc[26080](info): set up 2 sockets
Aug 10 22:33:10 dist-test-slave-njxd krb5kdc[26080](info): commencing operation
krb5kdc: starting...
W20260810 22:33:11.493323 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:11 dist-test-slave-njxd krb5kdc[26080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:33:11.619887 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:36671
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:36671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:11.732129 26094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:11.732436 26094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:11.732517 26094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:11.737147 26094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:11.737349 26094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:11.737393 26094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:11.737473 26094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:11.737510 26094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:11.743062 26094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:36671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:36671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:11.744143 26094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:11.745301 26094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:11.753501 26100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:11.753494 26101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:11.753492 26104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:11.753755 26094 server_base.cc:1034] running on GCE node
I20260810 22:33:11.754400 26094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:11.755466 26094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:11.756618 26094 hybrid_clock.cc:648] HybridClock initialized: now 1786401191756591 us; error 36 us; skew 500 ppm
Aug 10 22:33:11 dist-test-slave-njxd krb5kdc[26080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:11.759271 26094 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:33:11.760144 26094 webserver.cc:460] Webserver started at http://127.1.221.190:40647/ using document root <none> and password file <none>
I20260810 22:33:11.760692 26094 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:11.760777 26094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:11.760994 26094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:11.763038 26094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "a47d88c764c8404082bff86277f70d05"
format_stamp: "Formatted at 2026-08-10 22:33:11 on dist-test-slave-njxd"
server_key: "54bc227827f0b30987558c1368be4087"
server_key_iv: "ba7680db1e3e1df42a548c8e7187fb1f"
server_key_version: "encryptionkey@0"
I20260810 22:33:11.763576 26094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "a47d88c764c8404082bff86277f70d05"
format_stamp: "Formatted at 2026-08-10 22:33:11 on dist-test-slave-njxd"
server_key: "54bc227827f0b30987558c1368be4087"
server_key_iv: "ba7680db1e3e1df42a548c8e7187fb1f"
server_key_version: "encryptionkey@0"
I20260810 22:33:11.767086 26094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 22:33:11.769302 26112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:11.770466 26094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:33:11.770608 26094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "a47d88c764c8404082bff86277f70d05"
format_stamp: "Formatted at 2026-08-10 22:33:11 on dist-test-slave-njxd"
server_key: "54bc227827f0b30987558c1368be4087"
server_key_iv: "ba7680db1e3e1df42a548c8e7187fb1f"
server_key_version: "encryptionkey@0"
I20260810 22:33:11.770710 26094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:11.786855 26094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:11.789525 26094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:11.789786 26094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:11.797350 26094 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:36671
I20260810 22:33:11.797376 26167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:36671 every 8 connection(s)
I20260810 22:33:11.798345 26094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:11.801111 26168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:11.806663 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 26094
I20260810 22:33:11.806797 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:11.806763 26168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05: Bootstrap starting.
I20260810 22:33:11.807046 1910 external_mini_cluster.cc:1428] Setting key 7e9608520dda9923ad7fa63942946aad
I20260810 22:33:11.809540 26168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:11.810454 26168 log.cc:824] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:11.812716 26168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05: No bootstrap required, opened a new log
I20260810 22:33:11.816773 26168 raft_consensus.cc:348] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47d88c764c8404082bff86277f70d05" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36671 } }
Aug 10 22:33:11 dist-test-slave-njxd krb5kdc[26080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:33:11.817070 26168 raft_consensus.cc:374] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:11.817131 26168 raft_consensus.cc:729] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47d88c764c8404082bff86277f70d05, State: Initialized, Role: FOLLOWER
I20260810 22:33:11.817643 26168 consensus_queue.cc:260] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47d88c764c8404082bff86277f70d05" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36671 } }
I20260810 22:33:11.817817 26168 raft_consensus.cc:388] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:11.817895 26168 raft_consensus.cc:482] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:11.818005 26168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:11.819147 26168 raft_consensus.cc:504] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47d88c764c8404082bff86277f70d05" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36671 } }
I20260810 22:33:11.819519 26168 leader_election.cc:302] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a47d88c764c8404082bff86277f70d05; no voters:
I20260810 22:33:11.819859 26168 leader_election.cc:288] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:11.819942 26173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:11.820160 26173 raft_consensus.cc:686] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [term 1 LEADER]: Becoming Leader. State: Replica: a47d88c764c8404082bff86277f70d05, State: Running, Role: LEADER
I20260810 22:33:11.820575 26173 consensus_queue.cc:237] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47d88c764c8404082bff86277f70d05" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36671 } }
I20260810 22:33:11.821120 26168 sys_catalog.cc:564] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:11.822386 26174 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a47d88c764c8404082bff86277f70d05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47d88c764c8404082bff86277f70d05" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36671 } } }
I20260810 22:33:11.822532 26174 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:11.823807 26171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:11.808455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37323 (local address 127.1.221.190:36671)
0810 22:33:11.808893 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:11.808913 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:11.808954 (+ 41us) server_negotiation.cc:407] Connection header received
0810 22:33:11.809887 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:11.809913 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:11.810352 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:11.810726 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:11.811644 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:11.813060 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:11.814081 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:11.814818 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:11.814980 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:11.817665 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:11.817672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:11.817689 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:11.817719 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:11.819697 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:11.820327 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:11.820333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:11.820339 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:11.820483 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:11.820975 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:11.820979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:11.820980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:11.821455 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:11.821700 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:11.822474 (+ 774us) server_negotiation.cc:299] Negotiation successful
0810 22:33:11.822712 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":129,"threads_started":1}
I20260810 22:33:11.823002 26175 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47d88c764c8404082bff86277f70d05. Latest consensus state: current_term: 1 leader_uuid: "a47d88c764c8404082bff86277f70d05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a47d88c764c8404082bff86277f70d05" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 36671 } } }
I20260810 22:33:11.824147 26175 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05 [sys.catalog]: This master's current role is: LEADER
W20260810 22:33:11.826542 26190 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:33:11.826620 26190 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:33:11.826736 26182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:11.827085 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:11.827651 26182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:11.833563 26182 catalog_manager.cc:1347] Generated new cluster ID: faafd7d43bf14119925eb89dea47820c
I20260810 22:33:11.834740 26182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:11.838101 26171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:11.829338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60837 (local address 127.1.221.190:36671)
0810 22:33:11.829446 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:11.829450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:11.829647 (+ 197us) server_negotiation.cc:407] Connection header received
0810 22:33:11.829813 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:11.829817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:11.829858 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:11.829938 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:11.830557 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:11.831461 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:11.832611 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:11.832946 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:11.833060 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:11.834748 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:11.834750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:11.834752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:11.834773 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:11.836206 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:11.836741 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:11.836745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:11.836747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:11.836837 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:11.837152 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:11.837156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:11.837157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:11.837322 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:11.837470 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:11.837854 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 22:33:11.837947 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 22:33:11.857117 26182 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:11.858446 26182 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:11.866365 26182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a47d88c764c8404082bff86277f70d05: Generated new TSK 0
I20260810 22:33:11.867113 26182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:11.874766 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 26094
W20260810 22:33:11.881309 26078 connection.cc:535] client connection to 127.1.221.190:36671 recv error: Network error: failed to read from TLS socket (remote: 127.1.221.190:36671): Connection reset by peer (error 104)
2026-08-10T22:33:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:11 dist-test-slave-njxd krb5kdc[26202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:11 dist-test-slave-njxd krb5kdc[26202](info): set up 2 sockets
Aug 10 22:33:11 dist-test-slave-njxd krb5kdc[26202](info): commencing operation
krb5kdc: starting...
W20260810 22:33:12.967904 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:13 dist-test-slave-njxd krb5kdc[26202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:33:13.096261 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42553
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:42553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:13.209800 26216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:13.210093 26216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:13.210192 26216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:13.214174 26216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:13.214248 26216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:13.214287 26216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:13.214339 26216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:13.214380 26216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:13.219547 26216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:42553
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:13.220639 26216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:13.221714 26216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:13.230088 26224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:13.230099 26222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:13.230111 26226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:13.230376 26216 server_base.cc:1034] running on GCE node
I20260810 22:33:13.230960 26216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:13.232062 26216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:13.233243 26216 hybrid_clock.cc:648] HybridClock initialized: now 1786401193233218 us; error 44 us; skew 500 ppm
Aug 10 22:33:13 dist-test-slave-njxd krb5kdc[26202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401193, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:13.235934 26216 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:33:13.236738 26216 webserver.cc:460] Webserver started at http://127.1.221.190:42559/ using document root <none> and password file <none>
I20260810 22:33:13.237251 26216 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:13.237309 26216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:13.237468 26216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:13.239527 26216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "b9fdffe10b2a48d183175200b3292eb6"
format_stamp: "Formatted at 2026-08-10 22:33:13 on dist-test-slave-njxd"
server_key: "0556fbbae9a979453698844232f0fcf4"
server_key_iv: "f77f135cac813bf0aa1b7e0cba420efa"
server_key_version: "encryptionkey@0"
I20260810 22:33:13.240029 26216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "b9fdffe10b2a48d183175200b3292eb6"
format_stamp: "Formatted at 2026-08-10 22:33:13 on dist-test-slave-njxd"
server_key: "0556fbbae9a979453698844232f0fcf4"
server_key_iv: "f77f135cac813bf0aa1b7e0cba420efa"
server_key_version: "encryptionkey@0"
I20260810 22:33:13.243774 26216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:33:13.246025 26234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:13.247210 26216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:33:13.247332 26216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "b9fdffe10b2a48d183175200b3292eb6"
format_stamp: "Formatted at 2026-08-10 22:33:13 on dist-test-slave-njxd"
server_key: "0556fbbae9a979453698844232f0fcf4"
server_key_iv: "f77f135cac813bf0aa1b7e0cba420efa"
server_key_version: "encryptionkey@0"
I20260810 22:33:13.247432 26216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:13.264647 26216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:13.267459 26216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:13.267649 26216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:13.274727 26216 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:42553
I20260810 22:33:13.274760 26289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:42553 every 8 connection(s)
I20260810 22:33:13.275638 26216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:13.278209 26290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:13.282984 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 26216
I20260810 22:33:13.283113 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:13.283381 1910 external_mini_cluster.cc:1428] Setting key 2f7cd190c383536f1cb2ae6818dad6de
I20260810 22:33:13.283627 26290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6: Bootstrap starting.
I20260810 22:33:13.286043 26290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:13.286741 26290 log.cc:824] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:13.288518 26290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6: No bootstrap required, opened a new log
I20260810 22:33:13.291917 26293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:13.284616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55683 (local address 127.1.221.190:42553)
0810 22:33:13.285044 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:13.285058 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:13.285093 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:33:13.285797 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:13.285817 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:13.286161 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:13.286500 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:13.286922 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:13.287879 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:13.288799 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:13.289263 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:13.289442 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:13.290372 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:13.290378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:13.290390 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:13.290748 (+ 358us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:13.290799 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:13.290820 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:13.290957 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:13.291110 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 22:33:13.291248 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
I20260810 22:33:13.292068 26290 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fdffe10b2a48d183175200b3292eb6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42553 } }
I20260810 22:33:13.292304 26290 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:13.292377 26290 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fdffe10b2a48d183175200b3292eb6, State: Initialized, Role: FOLLOWER
I20260810 22:33:13.292891 26290 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fdffe10b2a48d183175200b3292eb6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42553 } }
I20260810 22:33:13.293046 26290 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:13.293111 26290 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:13.293202 26290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:13.294265 26290 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fdffe10b2a48d183175200b3292eb6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42553 } }
I20260810 22:33:13.294627 26290 leader_election.cc:302] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9fdffe10b2a48d183175200b3292eb6; no voters:
I20260810 22:33:13.294836 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:13.294974 26290 leader_election.cc:288] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:13.295081 26295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:13.295284 26295 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [term 1 LEADER]: Becoming Leader. State: Replica: b9fdffe10b2a48d183175200b3292eb6, State: Running, Role: LEADER
I20260810 22:33:13.295625 26295 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fdffe10b2a48d183175200b3292eb6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42553 } }
I20260810 22:33:13.296190 26290 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:13.296801 26295 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9fdffe10b2a48d183175200b3292eb6. Latest consensus state: current_term: 1 leader_uuid: "b9fdffe10b2a48d183175200b3292eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fdffe10b2a48d183175200b3292eb6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42553 } } }
I20260810 22:33:13.296927 26295 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:13.297220 26296 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9fdffe10b2a48d183175200b3292eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fdffe10b2a48d183175200b3292eb6" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42553 } } }
I20260810 22:33:13.297315 26296 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:13.297932 26306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:13.301162 26306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:13.302304 26293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:13.297120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60657 (local address 127.1.221.190:42553)
0810 22:33:13.297242 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:13.297246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:13.297798 (+ 552us) server_negotiation.cc:407] Connection header received
0810 22:33:13.297981 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:13.297984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:13.298027 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:13.298080 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:13.298504 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:13.299151 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:13.300336 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:13.300597 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:13.300703 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:13.301629 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:13.301632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:13.301635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:13.301647 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:13.301682 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:13.301696 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:13.301771 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:13.302103 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 22:33:13.302136 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:33:13.306859 26306 catalog_manager.cc:1347] Generated new cluster ID: 0856eed8e7cf4304b0bc072d0239c522
I20260810 22:33:13.306926 26306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:13.334188 26306 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:13.335434 26306 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:13.346529 26306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9fdffe10b2a48d183175200b3292eb6: Generated new TSK 0
I20260810 22:33:13.347318 26306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:13.359061 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 26216
W20260810 22:33:13.369287 26199 connection.cc:535] client connection to 127.1.221.190:42553 recv error: Network error: failed to read from TLS socket (remote: 127.1.221.190:42553): Connection reset by peer (error 104)
2026-08-10T22:33:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1487 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:13 dist-test-slave-njxd krb5kdc[26322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:13 dist-test-slave-njxd krb5kdc[26322](info): set up 2 sockets
Aug 10 22:33:13 dist-test-slave-njxd krb5kdc[26322](info): commencing operation
krb5kdc: starting...
W20260810 22:33:14.458770 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:14 dist-test-slave-njxd krb5kdc[26322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:33:14.585458 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42897
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:36783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:42897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:14.696133 26336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:14.696382 26336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:14.696421 26336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:14.700392 26336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:14.700444 26336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:14.700464 26336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:14.700480 26336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:14.700521 26336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:14.705210 26336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:36783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:42897
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:14.706214 26336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:14.707185 26336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:14.715344 26341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:14.715355 26344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:14.715436 26336 server_base.cc:1034] running on GCE node
W20260810 22:33:14.715355 26342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:14.716121 26336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:14.717150 26336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:14.718339 26336 hybrid_clock.cc:648] HybridClock initialized: now 1786401194718309 us; error 29 us; skew 500 ppm
Aug 10 22:33:14 dist-test-slave-njxd krb5kdc[26322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401194, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:14.724162 26336 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:33:14.724956 26336 webserver.cc:460] Webserver started at http://127.1.221.190:33761/ using document root <none> and password file <none>
I20260810 22:33:14.725452 26336 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:14.725507 26336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:14.725751 26336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:14.727686 26336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "2e982def8f254287924ea5c15a3ddcd4"
format_stamp: "Formatted at 2026-08-10 22:33:14 on dist-test-slave-njxd"
server_key: "be8d15649e5acd43c7627faf3a3595e4"
server_key_iv: "af82dc703967d8def947c56052f62a8f"
server_key_version: "encryptionkey@0"
I20260810 22:33:14.728188 26336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "2e982def8f254287924ea5c15a3ddcd4"
format_stamp: "Formatted at 2026-08-10 22:33:14 on dist-test-slave-njxd"
server_key: "be8d15649e5acd43c7627faf3a3595e4"
server_key_iv: "af82dc703967d8def947c56052f62a8f"
server_key_version: "encryptionkey@0"
I20260810 22:33:14.731822 26336 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:33:14.734141 26351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:14.735342 26336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:33:14.735437 26336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "2e982def8f254287924ea5c15a3ddcd4"
format_stamp: "Formatted at 2026-08-10 22:33:14 on dist-test-slave-njxd"
server_key: "be8d15649e5acd43c7627faf3a3595e4"
server_key_iv: "af82dc703967d8def947c56052f62a8f"
server_key_version: "encryptionkey@0"
I20260810 22:33:14.735519 26336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:14.739919 26336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:14.742833 26336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:14.742985 26336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:14.750000 26336 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:42897
I20260810 22:33:14.750028 26406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:42897 every 8 connection(s)
I20260810 22:33:14.750912 26336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:14.751848 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 26336
I20260810 22:33:14.752070 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:14.752403 1910 external_mini_cluster.cc:1428] Setting key 94a73f4eb470e769ed485585101fbfce
I20260810 22:33:14.754174 26407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:14.757886 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:14.753798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60611 (local address 127.1.221.190:42897)
0810 22:33:14.754191 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:14.754213 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:14.754258 (+ 45us) server_negotiation.cc:407] Connection header received
0810 22:33:14.755093 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:14.755115 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:14.755455 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:14.755823 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:14.756458 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:14.756463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:14.756478 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:14.756488 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:14.756544 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:14.756571 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:14.756721 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:14.756794 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 22:33:14.756940 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
I20260810 22:33:14.760836 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:14.760941 26407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4: Bootstrap starting.
I20260810 22:33:14.763401 26407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:14.764315 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:14.762794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54097 (local address 127.1.221.190:42897)
0810 22:33:14.762913 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:14.762918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:14.763120 (+ 202us) server_negotiation.cc:407] Connection header received
0810 22:33:14.763229 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:14.763231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:14.763255 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:14.763315 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:14.763961 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:14.763963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:14.763965 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:14.763975 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:14.763998 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:14.764007 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:14.764071 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:14.764179 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 22:33:14.764205 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:33:14.764309 26407 log.cc:824] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:14.766320 26407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4: No bootstrap required, opened a new log
I20260810 22:33:14.769644 26407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e982def8f254287924ea5c15a3ddcd4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42897 } }
I20260810 22:33:14.769923 26407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:14.770008 26407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e982def8f254287924ea5c15a3ddcd4, State: Initialized, Role: FOLLOWER
I20260810 22:33:14.770445 26407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e982def8f254287924ea5c15a3ddcd4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42897 } }
I20260810 22:33:14.770581 26407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:14.770649 26407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:14.770742 26407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:14.771564 26407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e982def8f254287924ea5c15a3ddcd4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42897 } }
I20260810 22:33:14.771888 26407 leader_election.cc:302] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e982def8f254287924ea5c15a3ddcd4; no voters:
I20260810 22:33:14.772194 26407 leader_election.cc:288] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:14.772329 26418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:14.772603 26418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [term 1 LEADER]: Becoming Leader. State: Replica: 2e982def8f254287924ea5c15a3ddcd4, State: Running, Role: LEADER
I20260810 22:33:14.772897 26418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e982def8f254287924ea5c15a3ddcd4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42897 } }
I20260810 22:33:14.773370 26407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:14.774621 26420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e982def8f254287924ea5c15a3ddcd4. Latest consensus state: current_term: 1 leader_uuid: "2e982def8f254287924ea5c15a3ddcd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e982def8f254287924ea5c15a3ddcd4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42897 } } }
I20260810 22:33:14.774744 26420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:14.775089 26419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e982def8f254287924ea5c15a3ddcd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e982def8f254287924ea5c15a3ddcd4" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42897 } } }
I20260810 22:33:14.775164 26430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:14.775183 26419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:14.778231 26430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:14.783270 26430 catalog_manager.cc:1347] Generated new cluster ID: a939b391bf0d407d8cc3403435cdd7f0
I20260810 22:33:14.783332 26430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:14.799558 26430 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:14.799949 26430 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:14.804498 26430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e982def8f254287924ea5c15a3ddcd4: Generated new TSK 0
I20260810 22:33:14.805276 26430 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:14.812757 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 26336
2026-08-10T22:33:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:14 dist-test-slave-njxd krb5kdc[26442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:14 dist-test-slave-njxd krb5kdc[26442](info): set up 2 sockets
Aug 10 22:33:14 dist-test-slave-njxd krb5kdc[26442](info): commencing operation
krb5kdc: starting...
W20260810 22:33:15.908797 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:15 dist-test-slave-njxd krb5kdc[26442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:33:16.033854 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44289
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:44289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:16.144116 26457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:16.144398 26457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:16.144470 26457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:16.148429 26457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:16.148522 26457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:16.148562 26457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:16.148617 26457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:16.148656 26457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:16.153846 26457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:44289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:16.154973 26457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:16.156030 26457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:16.164240 26463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:16.164251 26465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:16.164265 26462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:16.164477 26457 server_base.cc:1034] running on GCE node
I20260810 22:33:16.165021 26457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:16.166177 26457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:16.167359 26457 hybrid_clock.cc:648] HybridClock initialized: now 1786401196167344 us; error 26 us; skew 500 ppm
Aug 10 22:33:16 dist-test-slave-njxd krb5kdc[26442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:16.170192 26457 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:33:16.171072 26457 webserver.cc:460] Webserver started at http://127.1.221.190:38589/ using document root <none> and password file <none>
I20260810 22:33:16.171622 26457 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:16.171716 26457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:16.171947 26457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:16.173946 26457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "faa912c9c03e41c584e5e751aa1b360e"
format_stamp: "Formatted at 2026-08-10 22:33:16 on dist-test-slave-njxd"
server_key: "953d1d619835fed5f4ba8cecbdf55ee0"
server_key_iv: "965cd2d88cd53e7006e1921ee2c2f87d"
server_key_version: "encryptionkey@0"
I20260810 22:33:16.174491 26457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "faa912c9c03e41c584e5e751aa1b360e"
format_stamp: "Formatted at 2026-08-10 22:33:16 on dist-test-slave-njxd"
server_key: "953d1d619835fed5f4ba8cecbdf55ee0"
server_key_iv: "965cd2d88cd53e7006e1921ee2c2f87d"
server_key_version: "encryptionkey@0"
I20260810 22:33:16.178113 26457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:33:16.180440 26472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:16.181632 26457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:33:16.181789 26457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "faa912c9c03e41c584e5e751aa1b360e"
format_stamp: "Formatted at 2026-08-10 22:33:16 on dist-test-slave-njxd"
server_key: "953d1d619835fed5f4ba8cecbdf55ee0"
server_key_iv: "965cd2d88cd53e7006e1921ee2c2f87d"
server_key_version: "encryptionkey@0"
I20260810 22:33:16.181895 26457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:16.211254 26457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:16.214633 26457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:16.214843 26457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:16.222061 26457 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:44289
I20260810 22:33:16.222083 26532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:44289 every 8 connection(s)
I20260810 22:33:16.222992 26457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:16.225756 26533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:16.230597 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 26457
I20260810 22:33:16.230770 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:16.231055 1910 external_mini_cluster.cc:1428] Setting key bf17374bb21fd4ffde90a6c697df74ca
I20260810 22:33:16.231389 26533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e: Bootstrap starting.
I20260810 22:33:16.233942 26533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:16.234678 26533 log.cc:824] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:16.236582 26533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e: No bootstrap required, opened a new log
Aug 10 22:33:16 dist-test-slave-njxd krb5kdc[26442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:33:16.240348 26533 raft_consensus.cc:348] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa912c9c03e41c584e5e751aa1b360e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44289 } }
I20260810 22:33:16.240592 26533 raft_consensus.cc:374] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:16.240669 26533 raft_consensus.cc:729] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa912c9c03e41c584e5e751aa1b360e, State: Initialized, Role: FOLLOWER
I20260810 22:33:16.241086 26533 consensus_queue.cc:260] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa912c9c03e41c584e5e751aa1b360e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44289 } }
I20260810 22:33:16.241226 26533 raft_consensus.cc:388] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:16.241295 26533 raft_consensus.cc:482] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:16.241416 26533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:16.242308 26533 raft_consensus.cc:504] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa912c9c03e41c584e5e751aa1b360e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44289 } }
I20260810 22:33:16.242643 26533 leader_election.cc:302] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faa912c9c03e41c584e5e751aa1b360e; no voters:
I20260810 22:33:16.242980 26533 leader_election.cc:288] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:16.243062 26538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:16.243353 26538 raft_consensus.cc:686] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [term 1 LEADER]: Becoming Leader. State: Replica: faa912c9c03e41c584e5e751aa1b360e, State: Running, Role: LEADER
I20260810 22:33:16.243707 26538 consensus_queue.cc:237] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa912c9c03e41c584e5e751aa1b360e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44289 } }
I20260810 22:33:16.244307 26533 sys_catalog.cc:564] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:16.244868 26538 sys_catalog.cc:455] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [sys.catalog]: SysCatalogTable state changed. Reason: New leader faa912c9c03e41c584e5e751aa1b360e. Latest consensus state: current_term: 1 leader_uuid: "faa912c9c03e41c584e5e751aa1b360e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa912c9c03e41c584e5e751aa1b360e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44289 } } }
I20260810 22:33:16.245616 26538 sys_catalog.cc:458] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:16.245481 26539 sys_catalog.cc:455] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faa912c9c03e41c584e5e751aa1b360e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa912c9c03e41c584e5e751aa1b360e" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44289 } } }
I20260810 22:33:16.245977 26539 sys_catalog.cc:458] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:16.245978 26536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:16.232403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47927 (local address 127.1.221.190:44289)
0810 22:33:16.232865 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:16.232880 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:16.232915 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:33:16.233553 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:16.233570 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:16.233962 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:16.234312 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:16.235166 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:16.236158 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:16.237089 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:16.237562 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:16.237790 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:16.240478 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:16.240483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:16.240496 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:16.240522 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:16.242555 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:16.243060 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:16.243068 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:16.243076 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:16.243254 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:16.243585 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:16.243588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:16.243590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:16.244089 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:16.244273 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:16.244594 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 22:33:16.244900 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":153,"threads_started":1}
I20260810 22:33:16.247076 26547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:16.249233 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:16.250278 26547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:16.256991 26547 catalog_manager.cc:1347] Generated new cluster ID: 20b720d533454f41bc1c3f6827fad141
I20260810 22:33:16.257068 26547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:16.261298 26536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:16.251463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32781 (local address 127.1.221.190:44289)
0810 22:33:16.251580 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:16.251584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:16.251828 (+ 244us) server_negotiation.cc:407] Connection header received
0810 22:33:16.251969 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:16.251972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:16.252014 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:16.252098 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:16.254347 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:16.255311 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:16.256355 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:16.256715 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:16.256833 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:16.258185 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:16.258189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:16.258192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:16.258217 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:16.259633 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:16.260058 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:16.260062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:16.260064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:16.260145 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:16.260420 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:16.260425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:16.260427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:16.260601 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:16.260733 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:16.260982 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 22:33:16.261103 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:33:16.270339 26547 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:16.271288 26547 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:16.282159 26547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faa912c9c03e41c584e5e751aa1b360e: Generated new TSK 0
I20260810 22:33:16.282972 26547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:16.291029 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 26457
2026-08-10T22:33:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:16 dist-test-slave-njxd krb5kdc[26568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:16 dist-test-slave-njxd krb5kdc[26568](info): set up 2 sockets
Aug 10 22:33:16 dist-test-slave-njxd krb5kdc[26568](info): commencing operation
krb5kdc: starting...
W20260810 22:33:17.378929 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:17 dist-test-slave-njxd krb5kdc[26568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:33:17.508492 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:46741
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:41427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:46741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:17.631079 26582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:17.631385 26582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:17.631431 26582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:17.635988 26582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:17.636067 26582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:17.636087 26582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:17.636104 26582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:17.636121 26582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:17.642223 26582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:41427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:46741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:46741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:17.643460 26582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:17.644611 26582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:17.653699 26588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:17.653699 26590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:17.654012 26587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:17.653787 26582 server_base.cc:1034] running on GCE node
I20260810 22:33:17.654702 26582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:17.655973 26582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:17.657186 26582 hybrid_clock.cc:648] HybridClock initialized: now 1786401197657156 us; error 50 us; skew 500 ppm
Aug 10 22:33:17 dist-test-slave-njxd krb5kdc[26568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:17.660276 26582 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:33:17.661265 26582 webserver.cc:460] Webserver started at http://127.1.221.190:40169/ using document root <none> and password file <none>
I20260810 22:33:17.661947 26582 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:17.662047 26582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:17.662303 26582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:17.664640 26582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "fcef891b9f374b5196027214f99d5021"
format_stamp: "Formatted at 2026-08-10 22:33:17 on dist-test-slave-njxd"
server_key: "e98b562952a8919b5b9cd508dada5d0f"
server_key_iv: "ce35bf890a10fc13d2a3572be1334d5a"
server_key_version: "encryptionkey@0"
I20260810 22:33:17.665289 26582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "fcef891b9f374b5196027214f99d5021"
format_stamp: "Formatted at 2026-08-10 22:33:17 on dist-test-slave-njxd"
server_key: "e98b562952a8919b5b9cd508dada5d0f"
server_key_iv: "ce35bf890a10fc13d2a3572be1334d5a"
server_key_version: "encryptionkey@0"
I20260810 22:33:17.669550 26582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 22:33:17.672210 26597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:17.673558 26582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:33:17.673791 26582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "fcef891b9f374b5196027214f99d5021"
format_stamp: "Formatted at 2026-08-10 22:33:17 on dist-test-slave-njxd"
server_key: "e98b562952a8919b5b9cd508dada5d0f"
server_key_iv: "ce35bf890a10fc13d2a3572be1334d5a"
server_key_version: "encryptionkey@0"
I20260810 22:33:17.673918 26582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:17.696954 26582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:17.700292 26582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:17.700518 26582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:17.708463 26582 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:46741
I20260810 22:33:17.708496 26656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:46741 every 8 connection(s)
I20260810 22:33:17.709503 26582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:17.712412 26657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:17.717056 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 26582
I20260810 22:33:17.717211 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:17.717502 1910 external_mini_cluster.cc:1428] Setting key c3a17c037882bbb171b6ff22f0f07725
I20260810 22:33:17.718317 26657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021: Bootstrap starting.
I20260810 22:33:17.720682 26657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:17.721668 26657 log.cc:824] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:17.723842 26657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021: No bootstrap required, opened a new log
I20260810 22:33:17.727010 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:17.719110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55835 (local address 127.1.221.190:46741)
0810 22:33:17.719571 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:17.719588 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:17.719629 (+ 41us) server_negotiation.cc:407] Connection header received
0810 22:33:17.720342 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:17.720361 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:17.720721 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:17.721073 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:17.721426 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:17.722597 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:17.723769 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:17.724282 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:17.724406 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:17.725251 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:17.725257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:17.725270 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:17.725718 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:17.725774 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:17.725797 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:17.725890 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:17.726053 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 22:33:17.726283 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":150,"threads_started":1}
I20260810 22:33:17.727926 26657 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcef891b9f374b5196027214f99d5021" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46741 } }
I20260810 22:33:17.728243 26657 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:17.728317 26657 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcef891b9f374b5196027214f99d5021, State: Initialized, Role: FOLLOWER
I20260810 22:33:17.728775 26657 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcef891b9f374b5196027214f99d5021" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46741 } }
I20260810 22:33:17.728976 26657 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:17.729055 26657 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:17.729184 26657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:17.730088 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:17.730350 26657 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcef891b9f374b5196027214f99d5021" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46741 } }
I20260810 22:33:17.730693 26657 leader_election.cc:302] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcef891b9f374b5196027214f99d5021; no voters:
I20260810 22:33:17.731029 26657 leader_election.cc:288] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:17.731105 26663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:17.731320 26663 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [term 1 LEADER]: Becoming Leader. State: Replica: fcef891b9f374b5196027214f99d5021, State: Running, Role: LEADER
I20260810 22:33:17.731690 26663 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcef891b9f374b5196027214f99d5021" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46741 } }
I20260810 22:33:17.732355 26657 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:17.733284 26664 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcef891b9f374b5196027214f99d5021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcef891b9f374b5196027214f99d5021" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46741 } } }
I20260810 22:33:17.733528 26664 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:17.733846 26664 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcef891b9f374b5196027214f99d5021. Latest consensus state: current_term: 1 leader_uuid: "fcef891b9f374b5196027214f99d5021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcef891b9f374b5196027214f99d5021" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 46741 } } }
I20260810 22:33:17.733994 26664 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021 [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:17.733999 26677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:17.736837 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:17.732316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42027 (local address 127.1.221.190:46741)
0810 22:33:17.732432 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:17.732436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:17.732800 (+ 364us) server_negotiation.cc:407] Connection header received
0810 22:33:17.732925 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:17.732928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:17.732974 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:17.733053 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:17.733317 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:17.734072 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:17.735167 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:17.735489 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:17.735600 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:17.736481 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:17.736484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:17.736486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:17.736497 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:17.736521 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:17.736530 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:17.736585 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:17.736696 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 22:33:17.736740 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 22:33:17.737787 26677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:17.743328 26677 catalog_manager.cc:1347] Generated new cluster ID: e4087463fe8443e88dcb07aab981a9df
I20260810 22:33:17.743404 26677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:17.759635 26677 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:17.761016 26677 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:17.768674 26677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcef891b9f374b5196027214f99d5021: Generated new TSK 0
I20260810 22:33:17.769446 26677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:17.782562 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 26582
2026-08-10T22:33:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1490 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:17 dist-test-slave-njxd krb5kdc[26689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:17 dist-test-slave-njxd krb5kdc[26689](info): set up 2 sockets
Aug 10 22:33:17 dist-test-slave-njxd krb5kdc[26689](info): commencing operation
krb5kdc: starting...
W20260810 22:33:18.873840 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:18 dist-test-slave-njxd krb5kdc[26689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:33:19.004379 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42727
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:46683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:42727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:19.127637 26702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:19.127907 26702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:19.127986 26702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:19.132109 26702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:19.132171 26702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:19.132191 26702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:19.132246 26702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:19.132279 26702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:19.137785 26702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:46683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:42727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:42727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:19.138876 26702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:19.139922 26702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:19.148778 26710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:19.148779 26707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:19.148852 26708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:19.149080 26702 server_base.cc:1034] running on GCE node
I20260810 22:33:19.149824 26702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:19.150941 26702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:19.152132 26702 hybrid_clock.cc:648] HybridClock initialized: now 1786401199152100 us; error 46 us; skew 500 ppm
Aug 10 22:33:19 dist-test-slave-njxd krb5kdc[26689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:19.155000 26702 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:33:19.155934 26702 webserver.cc:460] Webserver started at http://127.1.221.190:34589/ using document root <none> and password file <none>
I20260810 22:33:19.156499 26702 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:19.156607 26702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:19.156864 26702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:19.159082 26702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "b9a3e3cef86143699415104c602405cd"
format_stamp: "Formatted at 2026-08-10 22:33:19 on dist-test-slave-njxd"
server_key: "88687548537e944b91b009c2678cc85d"
server_key_iv: "d2c24c3822c697c9c979e7942f26c355"
server_key_version: "encryptionkey@0"
I20260810 22:33:19.159806 26702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "b9a3e3cef86143699415104c602405cd"
format_stamp: "Formatted at 2026-08-10 22:33:19 on dist-test-slave-njxd"
server_key: "88687548537e944b91b009c2678cc85d"
server_key_iv: "d2c24c3822c697c9c979e7942f26c355"
server_key_version: "encryptionkey@0"
I20260810 22:33:19.163811 26702 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:33:19.166395 26717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:19.167676 26702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:33:19.167829 26702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "b9a3e3cef86143699415104c602405cd"
format_stamp: "Formatted at 2026-08-10 22:33:19 on dist-test-slave-njxd"
server_key: "88687548537e944b91b009c2678cc85d"
server_key_iv: "d2c24c3822c697c9c979e7942f26c355"
server_key_version: "encryptionkey@0"
I20260810 22:33:19.167944 26702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:19.172749 26702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:19.175706 26702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:19.175901 26702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:19.183403 26702 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:42727
I20260810 22:33:19.183414 26776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:42727 every 8 connection(s)
I20260810 22:33:19.184362 26702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:19.187291 26777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:19.191583 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 26702
I20260810 22:33:19.191731 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:19.192019 1910 external_mini_cluster.cc:1428] Setting key a2425f627954be61bb9a23e84da6e277
I20260810 22:33:19.193518 26777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd: Bootstrap starting.
I20260810 22:33:19.196398 26777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:19.197324 26777 log.cc:824] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:19.197844 26780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:19.193379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42949 (local address 127.1.221.190:42727)
0810 22:33:19.193858 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:19.193878 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:19.193921 (+ 43us) server_negotiation.cc:407] Connection header received
0810 22:33:19.194779 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:19.194808 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:19.195197 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:19.195555 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:19.196307 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:19.196315 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:19.196335 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:19.196352 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:19.196422 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:19.196455 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:19.196630 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:19.196741 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 22:33:19.196892 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":191,"threads_started":1}
I20260810 22:33:19.199343 26777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd: No bootstrap required, opened a new log
I20260810 22:33:19.200632 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:19.204015 26777 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a3e3cef86143699415104c602405cd" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42727 } }
I20260810 22:33:19.204308 26777 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:33:19.204383 26777 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a3e3cef86143699415104c602405cd, State: Initialized, Role: FOLLOWER
I20260810 22:33:19.204939 26780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:19.203098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51309 (local address 127.1.221.190:42727)
0810 22:33:19.203227 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:19.203231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:19.203468 (+ 237us) server_negotiation.cc:407] Connection header received
0810 22:33:19.203593 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:19.203595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:19.203642 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:19.203712 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:19.204447 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:19.204451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:19.204455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:33:19.204468 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:19.204495 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:33:19.204508 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:19.204576 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:19.204742 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 22:33:19.204799 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 22:33:19.204914 26777 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a3e3cef86143699415104c602405cd" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42727 } }
I20260810 22:33:19.205097 26777 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:33:19.205168 26777 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:33:19.205264 26777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:19.206215 26777 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a3e3cef86143699415104c602405cd" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42727 } }
I20260810 22:33:19.206540 26777 leader_election.cc:302] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9a3e3cef86143699415104c602405cd; no voters:
I20260810 22:33:19.206823 26777 leader_election.cc:288] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:33:19.206975 26787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:19.207278 26787 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [term 1 LEADER]: Becoming Leader. State: Replica: b9a3e3cef86143699415104c602405cd, State: Running, Role: LEADER
I20260810 22:33:19.207762 26787 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a3e3cef86143699415104c602405cd" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42727 } }
I20260810 22:33:19.207990 26777 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:19.208894 26787 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9a3e3cef86143699415104c602405cd. Latest consensus state: current_term: 1 leader_uuid: "b9a3e3cef86143699415104c602405cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a3e3cef86143699415104c602405cd" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42727 } } }
I20260810 22:33:19.209010 26787 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:19.209327 26788 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9a3e3cef86143699415104c602405cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a3e3cef86143699415104c602405cd" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 42727 } } }
I20260810 22:33:19.209453 26788 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:19.209393 26797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:19.212832 26797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:19.217886 26797 catalog_manager.cc:1347] Generated new cluster ID: 5a3f49db72b24c649527704fd56bda3d
I20260810 22:33:19.217953 26797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:19.238912 26797 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:19.239205 26797 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:19.245352 26797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9a3e3cef86143699415104c602405cd: Generated new TSK 0
I20260810 22:33:19.246042 26797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:33:19.256151 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 26702
2026-08-10T22:33:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1473 ms)
[----------] 18 tests from AuthTokenIssuingTest (26515 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:19 dist-test-slave-njxd krb5kdc[26812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:19 dist-test-slave-njxd krb5kdc[26812](info): set up 2 sockets
Aug 10 22:33:19 dist-test-slave-njxd krb5kdc[26812](info): commencing operation
krb5kdc: starting...
W20260810 22:33:20.708331 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.430s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:20 dist-test-slave-njxd krb5kdc[26812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:33:20.836650 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44259
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:44259,127.1.221.189:32985,127.1.221.188:35957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:20.949070 26825 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 22:33:20.949390 26825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:20.949460 26825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:20.949534 26825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:20.953353 26825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:20.953436 26825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:20.953472 26825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:20.953506 26825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:20.953538 26825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:20.958779 26825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:44259,127.1.221.189:32985,127.1.221.188:35957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:44259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:20.959826 26825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:20.960888 26825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:20.969417 26834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:20.969612 26825 server_base.cc:1034] running on GCE node
W20260810 22:33:20.969411 26831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:20.969411 26830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:20.970316 26825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:20.971419 26825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:20.972579 26825 hybrid_clock.cc:648] HybridClock initialized: now 1786401200972554 us; error 37 us; skew 500 ppm
Aug 10 22:33:20 dist-test-slave-njxd krb5kdc[26812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:20.979759 26825 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:33:20.980705 26825 webserver.cc:460] Webserver started at http://127.1.221.190:33895/ using document root <none> and password file <none>
I20260810 22:33:20.981267 26825 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:20.981361 26825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:20.981657 26825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:20.983706 26825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "4f1546f29b9c4f62bd0e02f6a99c500c"
format_stamp: "Formatted at 2026-08-10 22:33:20 on dist-test-slave-njxd"
server_key: "29b429da91128de1d6c31a4ceb9f2033"
server_key_iv: "86f14ca0f64f4b0f41f21c93b7989751"
server_key_version: "encryptionkey@0"
I20260810 22:33:20.984259 26825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "4f1546f29b9c4f62bd0e02f6a99c500c"
format_stamp: "Formatted at 2026-08-10 22:33:20 on dist-test-slave-njxd"
server_key: "29b429da91128de1d6c31a4ceb9f2033"
server_key_iv: "86f14ca0f64f4b0f41f21c93b7989751"
server_key_version: "encryptionkey@0"
I20260810 22:33:20.987955 26825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 22:33:20.990293 26843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:20.991523 26825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:33:20.991683 26825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "4f1546f29b9c4f62bd0e02f6a99c500c"
format_stamp: "Formatted at 2026-08-10 22:33:20 on dist-test-slave-njxd"
server_key: "29b429da91128de1d6c31a4ceb9f2033"
server_key_iv: "86f14ca0f64f4b0f41f21c93b7989751"
server_key_version: "encryptionkey@0"
I20260810 22:33:20.991791 26825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:21.001266 26825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:21.004495 26825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:21.004714 26825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:21.012094 26825 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:44259
I20260810 22:33:21.012394 26900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:44259 every 8 connection(s)
I20260810 22:33:21.013092 26825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:21.013316 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 26825
I20260810 22:33:21.013448 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:21.013859 1910 external_mini_cluster.cc:1428] Setting key 039e03f0bb38a7cbfce93066c1b50a19
I20260810 22:33:21.016029 26901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:21.021687 26901 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.221.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.189@KRBTEST.COM" created.
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:33:21.039120 26905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.025327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59080 (local address 127.1.221.190:44259)
0810 22:33:21.025661 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.025672 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.025710 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:33:21.026224 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.026232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.026432 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.026585 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.027733 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.029201 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.030788 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.031161 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.031287 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.034067 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.034073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.034084 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.034103 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.035839 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.036347 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.036352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.036353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.036455 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.036804 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.036807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.036808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.037183 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.037322 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.037742 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.038011 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":194,"thread_start_us":108,"threads_started":1}
I20260810 22:33:21.039120 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.024483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44259 (local address 127.0.0.1:59080)
0810 22:33:21.024998 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.025060 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:21.025915 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:21.026627 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:21.026632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:21.027161 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:21.027552 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.027568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.029361 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.029366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.030650 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.030665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.031339 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.031346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.031491 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.032105 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:21.032139 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:21.033915 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:21.035969 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.035978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.035989 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.036222 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.036579 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.036583 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.036585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.036706 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.037308 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:21.037315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:21.037456 (+ 141us) client_negotiation.cc:769] Sending connection context
0810 22:33:21.037647 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 22:33:21.038062 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":108,"thread_start_us":131,"threads_started":1}
I20260810 22:33:21.041970 26901 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:21.042820 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.042520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:32985 (local address 127.0.0.1:47936)
0810 22:33:21.042642 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.042730 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 22:33:21.043049 26846 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.189:32985: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.189.keytab.
Entry for principal kudu/127.1.221.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.189.keytab.
I20260810 22:33:21.044009 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.043739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:32985 (local address 127.0.0.1:47950)
0810 22:33:21.043863 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.043928 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 22:33:21.044168 26901 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:32985: Network error: Client connection negotiation failed: client connection to 127.1.221.189:32985: connect: Connection refused (error 111)
I20260810 22:33:21.044802 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:32985
--webserver_interface=127.1.221.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.189.keytab
--principal=kudu/127.1.221.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:44259,127.1.221.189:32985,127.1.221.188:35957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:33:21.093446 26901 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } attempt: 1
I20260810 22:33:21.094452 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.094140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:32985 (local address 127.0.0.1:47966)
0810 22:33:21.094305 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.094370 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 22:33:21.095257 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.094999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:32985 (local address 127.0.0.1:47968)
0810 22:33:21.095132 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.095206 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 22:33:21.095399 26901 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:32985: Network error: Client connection negotiation failed: client connection to 127.1.221.189:32985: connect: Connection refused (error 111)
W20260810 22:33:21.161383 26908 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 22:33:21.161712 26908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:21.161774 26908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:21.161859 26908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 22:33:21.163573 26901 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } attempt: 2
I20260810 22:33:21.164499 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.164158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:32985 (local address 127.0.0.1:47980)
0810 22:33:21.164347 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.164414 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 22:33:21.165334 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.165132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:32985 (local address 127.0.0.1:47990)
0810 22:33:21.165233 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.165280 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 22:33:21.165613 26901 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:32985: Network error: Client connection negotiation failed: client connection to 127.1.221.189:32985: connect: Connection refused (error 111)
W20260810 22:33:21.166543 26908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:21.166623 26908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:21.166687 26908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:21.166744 26908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:21.166782 26908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:21.175809 26908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:44259,127.1.221.189:32985,127.1.221.188:35957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.189.keytab
--principal=kudu/127.1.221.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:32985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.221.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:21.177587 26908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:21.179116 26908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:21.190093 26914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:21.190042 26918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:21.190033 26915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:21.190109 26908 server_base.cc:1034] running on GCE node
I20260810 22:33:21.191020 26908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:21.192562 26908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:21.193774 26908 hybrid_clock.cc:648] HybridClock initialized: now 1786401201193739 us; error 52 us; skew 500 ppm
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401201, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:21.196971 26908 init.cc:377] Logged in from keytab as kudu/127.1.221.189@KRBTEST.COM (short username kudu)
I20260810 22:33:21.198248 26908 webserver.cc:460] Webserver started at http://127.1.221.189:43153/ using document root <none> and password file <none>
I20260810 22:33:21.199083 26908 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:21.199178 26908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:21.199411 26908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:21.202351 26908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/data/instance:
uuid: "f5d5871a240b4f2b8ba5965c2501c5ad"
format_stamp: "Formatted at 2026-08-10 22:33:21 on dist-test-slave-njxd"
server_key: "34d7b2b2135657cac48d33b31e1e51a4"
server_key_iv: "6181197141b6984896c80321799691f6"
server_key_version: "encryptionkey@0"
I20260810 22:33:21.203111 26908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/wal/instance:
uuid: "f5d5871a240b4f2b8ba5965c2501c5ad"
format_stamp: "Formatted at 2026-08-10 22:33:21 on dist-test-slave-njxd"
server_key: "34d7b2b2135657cac48d33b31e1e51a4"
server_key_iv: "6181197141b6984896c80321799691f6"
server_key_version: "encryptionkey@0"
I20260810 22:33:21.208652 26908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 22:33:21.212244 26925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:21.213940 26908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 22:33:21.214092 26908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/wal
uuid: "f5d5871a240b4f2b8ba5965c2501c5ad"
format_stamp: "Formatted at 2026-08-10 22:33:21 on dist-test-slave-njxd"
server_key: "34d7b2b2135657cac48d33b31e1e51a4"
server_key_iv: "6181197141b6984896c80321799691f6"
server_key_version: "encryptionkey@0"
I20260810 22:33:21.214232 26908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:21.240597 26908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:21.243994 26908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:21.244180 26908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:21.251080 26908 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.189:32985
I20260810 22:33:21.251368 26979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.189:32985 every 8 connection(s)
I20260810 22:33:21.252084 26908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/data/info.pb
I20260810 22:33:21.254892 26980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:21.256804 26901 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } attempt: 3
I20260810 22:33:21.259195 26980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:21.262387 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 26908
I20260810 22:33:21.262549 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-1/wal/instance
I20260810 22:33:21.262887 1910 external_mini_cluster.cc:1428] Setting key 1efd9898397c7de0eea7199934347b8e
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401201, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:33:21.270823 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.257393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:32985 (local address 127.0.0.1:48006)
0810 22:33:21.257568 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.257609 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:21.257719 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:21.260051 (+ 2332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:21.260054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:21.260268 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:21.260428 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.260435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.261976 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.261980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.262827 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.262837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.263558 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.263561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.263674 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.264203 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:21.264219 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:21.265665 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:21.268417 (+ 2752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.268424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.268429 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.268727 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.269141 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.269145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.269147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.269234 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.270114 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:21.270119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:21.270264 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 22:33:21.270378 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 22:33:21.270623 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 22:33:21.272190 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.258206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.1.221.189:32985)
0810 22:33:21.258658 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.258681 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.258717 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:33:21.259377 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.259395 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.259773 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.260057 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.260577 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.261849 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.262944 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.263427 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.263591 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.265821 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.265830 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.265846 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.265884 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.268282 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.268885 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.268894 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.268901 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.269038 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.269337 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.269344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.269349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.269961 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.270594 (+ 633us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.270782 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.271088 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":141,"threads_started":1}
I20260810 22:33:21.272841 26905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.261208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59084 (local address 127.1.221.190:44259)
0810 22:33:21.261347 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.261352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.261621 (+ 269us) server_negotiation.cc:407] Connection header received
0810 22:33:21.261788 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.261791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.261835 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.261932 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.262774 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.263503 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.264938 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.265318 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.265420 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.268193 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.268197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.268199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.268223 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.270608 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.271375 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.271379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.271381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.271467 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.271756 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.271759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.271761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.271924 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.272026 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.272595 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.272709 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.1.221.188@KRBTEST.COM; defaulting to no policy
I20260810 22:33:21.273504 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.261141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44259 (local address 127.0.0.1:59084)
0810 22:33:21.261477 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.261529 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:21.261689 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:21.262008 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:21.262014 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:21.262500 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:21.262644 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.262651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.263710 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.263716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.264780 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.264804 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.265603 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.265612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.265746 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.266315 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:21.266336 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:21.268055 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:21.270732 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.270742 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.270748 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.271270 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.271576 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.271579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.271581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.271660 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.272030 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:21.272038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:21.272149 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 22:33:21.273169 (+ 1020us) client_negotiation.cc:240] Negotiation successful
0810 22:33:21.273386 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":97,"threads_started":1}
I20260810 22:33:21.274351 26901 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:21.274848 26980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:21.275146 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.274895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:50396)
0810 22:33:21.275022 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.275079 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 22:33:21.275955 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.275658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:50408)
0810 22:33:21.275837 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.275891 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260810 22:33:21.276095 26901 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:35957: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.188@KRBTEST.COM" created.
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401201, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260810 22:33:21.284982 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.275519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.1.221.189:32985)
0810 22:33:21.275629 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.275635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.275661 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:33:21.275690 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.275693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.275730 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.275798 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.276305 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.276870 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.278060 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.278419 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.278543 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.280701 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.280708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.280714 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.280747 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.282548 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.283073 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.283080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.283084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.283256 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.283959 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.283963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.283964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.284234 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.284337 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.284672 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.284800 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:33:21.286026 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.275355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:32985 (local address 127.0.0.1:48020)
0810 22:33:21.275461 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.275475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:21.275596 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:21.275892 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:21.275895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:21.276070 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:21.276191 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.276202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.277011 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.277018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.277935 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.277950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.278542 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.278545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.278638 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.279120 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:21.279134 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:21.280552 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:21.282692 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.282695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.282696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.282953 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.283733 (+ 780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.283738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.283740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.283838 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.284364 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:21.284367 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:21.284471 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 22:33:21.284605 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 22:33:21.285792 (+ 1187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":1}
I20260810 22:33:21.287124 26980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:21.287983 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.287647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:50414)
0810 22:33:21.287807 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.287912 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 22:33:21.288640 26928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
I20260810 22:33:21.289796 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.289492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:50428)
0810 22:33:21.289660 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.289712 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 22:33:21.290058 26980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:35957: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.188.keytab.
Entry for principal kudu/127.1.221.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.188.keytab.
I20260810 22:33:21.296152 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:35957
--webserver_interface=127.1.221.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.188.keytab
--principal=kudu/127.1.221.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:44259,127.1.221.189:32985,127.1.221.188:35957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:33:21.307297 26901 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } attempt: 1
I20260810 22:33:21.308043 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.307784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:50444)
0810 22:33:21.307918 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.307973 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 22:33:21.308718 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.308511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:50452)
0810 22:33:21.308616 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.308667 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 22:33:21.308872 26901 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:35957: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
I20260810 22:33:21.339380 26980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } attempt: 1
I20260810 22:33:21.340406 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.340068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:50454)
0810 22:33:21.340246 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.340323 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 22:33:21.341248 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.341023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:50470)
0810 22:33:21.341133 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.341189 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 22:33:21.341405 26980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:35957: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
I20260810 22:33:21.384016 26901 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } attempt: 2
I20260810 22:33:21.384927 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.384556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:50474)
0810 22:33:21.384754 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.384831 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 22:33:21.385812 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.385587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:50486)
0810 22:33:21.385710 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.385759 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 22:33:21.386042 26901 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:35957: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
I20260810 22:33:21.409559 26980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } attempt: 2
I20260810 22:33:21.410398 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.410106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:57120)
0810 22:33:21.410253 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.410321 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 22:33:21.411145 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.410954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:57134)
0810 22:33:21.411052 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.411096 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 22:33:21.411340 26980 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:35957: Network error: Client connection negotiation failed: client connection to 127.1.221.188:35957: connect: Connection refused (error 111)
W20260810 22:33:21.412912 26987 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 22:33:21.413194 26987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:21.413245 26987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:21.413331 26987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:21.417359 26987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:21.417412 26987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:21.417459 26987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:21.417495 26987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:21.417527 26987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:21.422956 26987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:44259,127.1.221.189:32985,127.1.221.188:35957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.188.keytab
--principal=kudu/127.1.221.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:35957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.221.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:21.424100 26987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:21.425123 26987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:21.433806 26993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:21.433804 26996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:21.433904 26987 server_base.cc:1034] running on GCE node
W20260810 22:33:21.433804 26994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:21.434751 26987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:21.435925 26987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:21.437124 26987 hybrid_clock.cc:648] HybridClock initialized: now 1786401201437111 us; error 31 us; skew 500 ppm
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401201, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:21.440044 26987 init.cc:377] Logged in from keytab as kudu/127.1.221.188@KRBTEST.COM (short username kudu)
I20260810 22:33:21.441035 26987 webserver.cc:460] Webserver started at http://127.1.221.188:40105/ using document root <none> and password file <none>
I20260810 22:33:21.441699 26987 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:21.441809 26987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:21.442082 26987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:21.444283 26987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/data/instance:
uuid: "bffda888a43f4aaabdad8a41e9c7b5d0"
format_stamp: "Formatted at 2026-08-10 22:33:21 on dist-test-slave-njxd"
server_key: "deb6b57936396ab7993694102696d6d9"
server_key_iv: "953e98156a1476ab01dd1e5de3a60172"
server_key_version: "encryptionkey@0"
I20260810 22:33:21.444871 26987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/wal/instance:
uuid: "bffda888a43f4aaabdad8a41e9c7b5d0"
format_stamp: "Formatted at 2026-08-10 22:33:21 on dist-test-slave-njxd"
server_key: "deb6b57936396ab7993694102696d6d9"
server_key_iv: "953e98156a1476ab01dd1e5de3a60172"
server_key_version: "encryptionkey@0"
I20260810 22:33:21.448662 26987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:33:21.451136 27003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:21.452360 26987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:33:21.452507 26987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/wal
uuid: "bffda888a43f4aaabdad8a41e9c7b5d0"
format_stamp: "Formatted at 2026-08-10 22:33:21 on dist-test-slave-njxd"
server_key: "deb6b57936396ab7993694102696d6d9"
server_key_iv: "953e98156a1476ab01dd1e5de3a60172"
server_key_version: "encryptionkey@0"
I20260810 22:33:21.452617 26987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:21.472410 26987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:21.475883 26987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:21.476107 26987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:21.483628 26987 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.188:35957
I20260810 22:33:21.483940 27062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.188:35957 every 8 connection(s)
I20260810 22:33:21.484647 26987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/data/info.pb
I20260810 22:33:21.485221 26901 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } attempt: 3
I20260810 22:33:21.487995 27063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:21.492564 27063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:21.493340 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 26987
I20260810 22:33:21.493466 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786400913154531-1910-0/minicluster-data/master-2/wal/instance
I20260810 22:33:21.493794 1910 external_mini_cluster.cc:1428] Setting key f49c9f531c13409db31cbe3a0cbcfcf3
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401200, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260810 22:33:21.499233 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.485831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:57140)
0810 22:33:21.485996 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.486013 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:21.486118 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:21.488604 (+ 2486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:21.488607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:21.488779 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:21.488882 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.488887 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.490726 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.490729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.491253 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.491261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.492199 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.492201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.492266 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.492786 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:21.492799 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:21.494349 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:21.497202 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.497206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.497208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.497478 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.497864 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.497867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.497869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.497988 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.498691 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:21.498694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:21.498790 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 22:33:21.498901 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 22:33:21.499048 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 22:33:21.500881 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.486484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57140 (local address 127.1.221.188:35957)
0810 22:33:21.486900 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.486924 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.486967 (+ 43us) server_negotiation.cc:407] Connection header received
0810 22:33:21.487847 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.487871 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.488275 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.488637 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.489056 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.490576 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.491372 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.492065 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.492223 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.494584 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.494592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.494608 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.494650 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.497084 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.497605 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.497613 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.497639 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.497758 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.498080 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.498084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.498086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.498574 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.498721 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.499727 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.500026 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":121,"threads_started":1}
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260810 22:33:21.502530 26980 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } attempt: 3
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401201, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:33:21.505867 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.494420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.1.221.188:35957)
0810 22:33:21.494789 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.494799 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.495087 (+ 288us) server_negotiation.cc:407] Connection header received
0810 22:33:21.495131 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.495134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.495179 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.495257 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.496307 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.497084 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.498349 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.498752 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.498871 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.501685 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.501689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.501691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.501730 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.503568 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.504037 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.504040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.504041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.504125 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.504453 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.504460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.504464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.504679 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.504855 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.505521 (+ 666us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.505658 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"spinlock_wait_cycles":2816,"thread_start_us":140,"threads_started":1}
I20260810 22:33:21.506107 26901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c: Bootstrap starting.
I20260810 22:33:21.508486 26905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.494973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.1.221.190:44259)
0810 22:33:21.495091 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.495096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.495495 (+ 399us) server_negotiation.cc:407] Connection header received
0810 22:33:21.495702 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.495706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.495755 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.495852 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.496770 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.497321 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.498536 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.498878 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.498963 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.504464 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.504467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.504469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.504494 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.506315 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.506974 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.506977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.506979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.507072 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.507403 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.507406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.507408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.507665 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.507740 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.508189 (+ 449us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.508309 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:33:21.509056 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.494520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:44259 (local address 127.0.0.1:34260)
0810 22:33:21.495372 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.495424 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:21.495582 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:21.496105 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:21.496112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:21.496532 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:21.496648 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.496655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.497435 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.497438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.498422 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.498433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.499077 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.499081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.499204 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.501734 (+ 2530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:21.501748 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:21.504333 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:21.506527 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.506539 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.506548 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.506843 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.507169 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.507179 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.507180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.507294 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.507830 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:21.507836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:21.507955 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 22:33:21.508627 (+ 672us) client_negotiation.cc:240] Negotiation successful
0810 22:33:21.508833 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"spinlock_wait_cycles":768,"thread_start_us":132,"threads_started":1}
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401201, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260810 22:33:21.509575 26901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:21.510807 27063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:21.511118 26901 log.cc:824] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:21.513360 26901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c: No bootstrap required, opened a new log
I20260810 22:33:21.513656 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.503304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:57146)
0810 22:33:21.504561 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.504578 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:21.504702 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:21.505163 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:21.505168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:21.505358 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:21.505500 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.505510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.506626 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.506629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.507363 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.507375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.507974 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.507977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.508058 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.508484 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:21.508498 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:21.509920 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:21.511997 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.512002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.512004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.512268 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.512572 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.512575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.512577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.512675 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.513070 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:21.513073 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:21.513234 (+ 161us) client_negotiation.cc:769] Sending connection context
0810 22:33:21.513316 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 22:33:21.513456 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160}
I20260810 22:33:21.514456 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.503549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.1.221.188:35957)
0810 22:33:21.504921 (+ 1372us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.504924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.504937 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:33:21.504972 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.504975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.505016 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.505632 (+ 616us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.505743 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.506508 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.507503 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.507839 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.508281 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.510082 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.510088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.510094 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.510123 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.511863 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.512388 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.512390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.512392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.512470 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.512789 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.512792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.512794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.512971 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.514132 (+ 1161us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.514221 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.514325 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1275}
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401201, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260810 22:33:21.518133 26980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad: Bootstrap starting.
I20260810 22:33:21.518297 26901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } }
I20260810 22:33:21.518534 26901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f1546f29b9c4f62bd0e02f6a99c500c, State: Initialized, Role: FOLLOWER
I20260810 22:33:21.518896 26901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } }
I20260810 22:33:21.520402 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.511550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.1.221.189:32985)
0810 22:33:21.511679 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.511684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.511699 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:33:21.511737 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.511742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.511781 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.511870 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.512362 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.513076 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.514197 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.514514 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.514625 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.516878 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.516885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.516891 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.516920 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.518531 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.519051 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.519055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.519056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.519140 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.519458 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.519463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.519466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.519721 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.519797 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.520134 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.520245 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 22:33:21.520393 26901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:21.520843 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.511301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:32985 (local address 127.0.0.1:35426)
0810 22:33:21.511545 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.511560 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:21.511648 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:21.511899 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:21.511905 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:21.512097 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:21.512220 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.512230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.513244 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.513247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.514058 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.514071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.514785 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.514788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.514880 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.515324 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:21.515338 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:21.516721 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:21.518680 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.518683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.518684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.518933 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.519257 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.519259 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.519261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.519344 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.519873 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:21.519876 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:21.519960 (+ 84us) client_negotiation.cc:769] Sending connection context
0810 22:33:21.520554 (+ 594us) client_negotiation.cc:240] Negotiation successful
0810 22:33:21.520695 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260810 22:33:21.520959 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.507644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.1.221.190:44259)
0810 22:33:21.508194 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.508214 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.508225 (+ 11us) server_negotiation.cc:407] Connection header received
0810 22:33:21.508270 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.508276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.508316 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.508440 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.510165 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.510959 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.512576 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.512903 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.516014 (+ 3111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.516130 (+ 116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.516133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.516136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.516153 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.518436 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.518939 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.518942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.518943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.519026 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.519381 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.519384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.519386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.519578 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.520436 (+ 858us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.520603 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.520724 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":95,"threads_started":1}
I20260810 22:33:21.521046 26980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:21.521732 27063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:21.521996 26980 log.cc:824] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:21.525265 26980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad: No bootstrap required, opened a new log
W20260810 22:33:21.526592 27084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:33:21.526670 27084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:33:21.525537 26825 master_runner.cc:186] Error getting master registration for 127.1.221.189:32985: OK, instance_id { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" instance_seqno: 1786401201249201 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 22:33:21.529184 26825 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 22:33:21.529481 26980 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } }
I20260810 22:33:21.529946 26980 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d5871a240b4f2b8ba5965c2501c5ad, State: Initialized, Role: FOLLOWER
I20260810 22:33:21.530354 26980 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } }
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260810 22:33:21.531565 26980 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 22:33:21 dist-test-slave-njxd krb5kdc[26812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401201, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260810 22:33:21.536696 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.522613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.1.221.189:32985)
0810 22:33:21.522714 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.522718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.522989 (+ 271us) server_negotiation.cc:407] Connection header received
0810 22:33:21.523029 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.523033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.523082 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.523152 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.524015 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.524700 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.525778 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.526098 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.526176 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.532508 (+ 6332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.532511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.532514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.532542 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.534710 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.535251 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.535255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.535257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.535348 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.535671 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.535675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.535677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.535884 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.536045 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.536439 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.536550 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 22:33:21.537034 27100 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:33:21.537103 27100 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:33:21.537329 27088 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } } }
I20260810 22:33:21.537437 27088 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:33:21.537745 27071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } } }
I20260810 22:33:21.537889 27071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:33:21.541640 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:21.543083 26871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34264
I20260810 22:33:21.543385 26871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 22:33:21.543510 26871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:21.544907 26871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } }
I20260810 22:33:21.545905 26871 leader_election.cc:288] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [CANDIDATE]: Term 1 election: Requested vote from peers f5d5871a240b4f2b8ba5965c2501c5ad (127.1.221.189:32985), bffda888a43f4aaabdad8a41e9c7b5d0 (127.1.221.188:35957)
I20260810 22:33:21.547241 26954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad"
I20260810 22:33:21.547485 26954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:21.548878 26954 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f1546f29b9c4f62bd0e02f6a99c500c in term 1.
I20260810 22:33:21.545641 26908 master_runner.cc:186] Error getting master registration for 127.1.221.188:35957: OK, instance_id { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" instance_seqno: 1786401201481486 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 22:33:21.549345 26846 leader_election.cc:302] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f1546f29b9c4f62bd0e02f6a99c500c, f5d5871a240b4f2b8ba5965c2501c5ad; no voters:
I20260810 22:33:21.549456 26908 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 22:33:21.549652 27071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:21.549870 27071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [term 1 LEADER]: Becoming Leader. State: Replica: 4f1546f29b9c4f62bd0e02f6a99c500c, State: Running, Role: LEADER
I20260810 22:33:21.550208 27071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } }
I20260810 22:33:21.552026 27071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f1546f29b9c4f62bd0e02f6a99c500c. Latest consensus state: current_term: 1 leader_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } } }
I20260810 22:33:21.552130 27071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:21.552534 27108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:21.553371 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.522221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:35957 (local address 127.0.0.1:57152)
0810 22:33:21.526638 (+ 4417us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:21.526652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:21.526761 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:21.527038 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:21.527042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:21.527211 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:21.527288 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.527294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.528237 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.528240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.529008 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:21.529018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.529658 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.529662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:21.529744 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.530192 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:21.530206 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:21.532265 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:21.549737 (+ 17472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.549741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.549743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.550048 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.552113 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:21.552126 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:21.552128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:21.552231 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:21.552863 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:21.552866 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:21.552950 (+ 84us) client_negotiation.cc:769] Sending connection context
0810 22:33:21.553036 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 22:33:21.553195 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4345}
I20260810 22:33:21.553498 27108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:33:21.554183 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.522522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.1.221.188:35957)
0810 22:33:21.523194 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.523198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.526710 (+ 3512us) server_negotiation.cc:407] Connection header received
0810 22:33:21.526843 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.526845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.526880 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.526921 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.527402 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.528118 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.529133 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.529423 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.529503 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.532412 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.532416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.532418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.532447 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.534176 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.550325 (+ 16149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.550329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.550330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.550425 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.552431 (+ 2006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.552447 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.552449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.552674 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.552764 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.553917 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.554037 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589}
I20260810 22:33:21.557180 27063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0: Bootstrap starting.
I20260810 22:33:21.559973 27063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:21.560830 27063 log.cc:824] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:21.563431 27063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0: No bootstrap required, opened a new log
I20260810 22:33:21.568022 27063 raft_consensus.cc:348] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } }
I20260810 22:33:21.568266 27063 raft_consensus.cc:729] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffda888a43f4aaabdad8a41e9c7b5d0, State: Initialized, Role: FOLLOWER
I20260810 22:33:21.568763 27063 consensus_queue.cc:260] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } }
I20260810 22:33:21.570183 27063 sys_catalog.cc:564] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:21.570688 27033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0"
I20260810 22:33:21.570919 27033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:21.572791 27033 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f1546f29b9c4f62bd0e02f6a99c500c in term 1.
I20260810 22:33:21.578881 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.556878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.1.221.188:35957)
0810 22:33:21.557144 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.557148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.564146 (+ 6998us) server_negotiation.cc:407] Connection header received
0810 22:33:21.564379 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.564382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.564425 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.565509 (+ 1084us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.567207 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.567962 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.570137 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.570476 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.570548 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.575134 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.575138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.575141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.575168 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.576893 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.577405 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.577409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.577411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.577498 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.577820 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.577824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.577825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.578036 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.578182 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.578572 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.578695 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260810 22:33:21.584715 27033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0 [term 1 FOLLOWER]: Refusing update from remote peer 4f1546f29b9c4f62bd0e02f6a99c500c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:21.584905 26954 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad [term 1 FOLLOWER]: Refusing update from remote peer 4f1546f29b9c4f62bd0e02f6a99c500c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:21.585973 27102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:21.586701 27071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:21.587585 27114 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } } }
I20260810 22:33:21.587790 27114 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:33:21.589758 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.561447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.1.221.190:44259)
0810 22:33:21.561689 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.561693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.563741 (+ 2048us) server_negotiation.cc:407] Connection header received
0810 22:33:21.563787 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.563791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.563842 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.563925 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.573084 (+ 9159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.573932 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.575486 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.575866 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.575942 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.582345 (+ 6403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.582349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.582352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.582385 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.584167 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.588301 (+ 4134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.588305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.588307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.588403 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.588723 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.588727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.588729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.588955 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.589066 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.589430 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.589596 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260810 22:33:21.596489 27088 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f1546f29b9c4f62bd0e02f6a99c500c. Latest consensus state: current_term: 1 leader_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } } }
I20260810 22:33:21.596633 27088 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:33:21.596956 27114 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f1546f29b9c4f62bd0e02f6a99c500c. Latest consensus state: current_term: 1 leader_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } } }
I20260810 22:33:21.597543 27114 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 22:33:21.598249 27126 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:33:21.598327 27126 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:33:21.599305 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:21.561301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.1.221.189:32985)
0810 22:33:21.561569 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.561593 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.563700 (+ 2107us) server_negotiation.cc:407] Connection header received
0810 22:33:21.564050 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.564053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.564100 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.564170 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.577360 (+ 13190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.578203 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.579390 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:21.579742 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:21.579819 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:21.591445 (+ 11626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:21.591449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:21.591452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:21.591486 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:21.596680 (+ 5194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.597818 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.597823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.597825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.597918 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:21.598235 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:21.598238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:21.598240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:21.598467 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:21.598580 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:21.599008 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 22:33:21.599149 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260810 22:33:21.602828 27071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } } }
I20260810 22:33:21.602968 27071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:21.603727 27102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } } }
I20260810 22:33:21.603844 27102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:21.604614 27114 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } } }
I20260810 22:33:21.604738 27114 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:33:21.604959 27088 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f1546f29b9c4f62bd0e02f6a99c500c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 44259 } } peers { permanent_uuid: "f5d5871a240b4f2b8ba5965c2501c5ad" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 32985 } } peers { permanent_uuid: "bffda888a43f4aaabdad8a41e9c7b5d0" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 35957 } } }
I20260810 22:33:21.605036 27127 mvcc.cc:205] Tried to move back new op lower bound from 7317099321680826368 to 7317099321553207296. Current Snapshot: MvccSnapshot[applied={T|T < 7317099321680826368}]
I20260810 22:33:21.605098 27088 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:33:21.606619 27113 mvcc.cc:205] Tried to move back new op lower bound from 7317099321680826368 to 7317099321553207296. Current Snapshot: MvccSnapshot[applied={T|T < 7317099321680826368}]
I20260810 22:33:21.606805 27129 mvcc.cc:205] Tried to move back new op lower bound from 7317099321680826368 to 7317099321553207296. Current Snapshot: MvccSnapshot[applied={T|T < 7317099321680826368}]
I20260810 22:33:21.610177 27108 catalog_manager.cc:1347] Generated new cluster ID: b93f33947c3e420eb342c74fe979acd5
I20260810 22:33:21.610248 27108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:21.625052 27108 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:21.626430 27108 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:21.643999 27108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f1546f29b9c4f62bd0e02f6a99c500c: Generated new TSK 0
I20260810 22:33:21.644997 27108 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:33:21.679105 27069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:21.675806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34272 (local address 127.1.221.190:44259)
0810 22:33:21.675943 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.675949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.677958 (+ 2009us) server_negotiation.cc:407] Connection header received
0810 22:33:21.678161 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.678165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.678236 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.678290 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.678965 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34272: BlockingRecv error: recv got EOF from 127.0.0.1:34272 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 22:33:21.680503 26982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:21.676619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.1.221.189:32985)
0810 22:33:21.676734 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.676738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.679614 (+ 2876us) server_negotiation.cc:407] Connection header received
0810 22:33:21.679790 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.679794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.679843 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.679893 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.680382 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35450: BlockingRecv error: recv got EOF from 127.0.0.1:35450 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 22:33:21.681923 27065 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:21.677273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.1.221.188:35957)
0810 22:33:21.677388 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.677393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.680954 (+ 3561us) server_negotiation.cc:407] Connection header received
0810 22:33:21.681117 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.681120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.681167 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.681231 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.681803 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57170: BlockingRecv error: recv got EOF from 127.0.0.1:57170 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 22:33:21.686555 26982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:21.684901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.1.221.189:32985)
0810 22:33:21.685004 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.685009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.685541 (+ 532us) server_negotiation.cc:407] Connection header received
0810 22:33:21.685835 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.685838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.685910 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.685961 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.686458 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35464: BlockingRecv error: recv got EOF from 127.0.0.1:35464 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 22:33:21.691601 26982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:21.690052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.1.221.189:32985)
0810 22:33:21.690153 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.690157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.690709 (+ 552us) server_negotiation.cc:407] Connection header received
0810 22:33:21.690896 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.690900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.690949 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.690998 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.691507 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35476: BlockingRecv error: recv got EOF from 127.0.0.1:35476 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 22:33:21.697443 26982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:21.695805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.1.221.189:32985)
0810 22:33:21.695908 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.695913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.696493 (+ 580us) server_negotiation.cc:407] Connection header received
0810 22:33:21.696699 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.696702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.696755 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.696807 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.697340 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35486: BlockingRecv error: recv got EOF from 127.0.0.1:35486 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 22:33:21.705240 26982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:21.703512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.1.221.189:32985)
0810 22:33:21.703652 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.703657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.704279 (+ 622us) server_negotiation.cc:407] Connection header received
0810 22:33:21.704474 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.704478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.704530 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.704581 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.705122 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35494: BlockingRecv error: recv got EOF from 127.0.0.1:35494 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 22:33:21.718264 26982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:21.716230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.1.221.189:32985)
0810 22:33:21.716356 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.716361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.717074 (+ 713us) server_negotiation.cc:407] Connection header received
0810 22:33:21.717325 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.717329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.717386 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.717458 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.718141 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35496: BlockingRecv error: recv got EOF from 127.0.0.1:35496 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 22:33:21.738934 26982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:21.736931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.1.221.189:32985)
0810 22:33:21.737076 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.737081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.737765 (+ 684us) server_negotiation.cc:407] Connection header received
0810 22:33:21.737989 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.737993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.738051 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.738106 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.738806 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35500: BlockingRecv error: recv got EOF from 127.0.0.1:35500 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 22:33:21.775068 26982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:21.773823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.1.221.189:32985)
0810 22:33:21.773972 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.773977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.774122 (+ 145us) server_negotiation.cc:407] Connection header received
0810 22:33:21.774249 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.774253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.774319 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.774436 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.774941 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35516: BlockingRecv error: recv got EOF from 127.0.0.1:35516 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 22:33:21.842440 26982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:21.841251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.1.221.189:32985)
0810 22:33:21.841403 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.841407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.841564 (+ 157us) server_negotiation.cc:407] Connection header received
0810 22:33:21.841762 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.841765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.841817 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.841902 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.842338 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35532: BlockingRecv error: recv got EOF from 127.0.0.1:35532 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 22:33:21.973956 26982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:21.972788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.1.221.189:32985)
0810 22:33:21.972965 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:21.972969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:21.973076 (+ 107us) server_negotiation.cc:407] Connection header received
0810 22:33:21.973206 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:21.973209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:21.973260 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:21.973333 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:21.973825 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35540: BlockingRecv error: recv got EOF from 127.0.0.1:35540 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 22:33:22.233270 26982 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:22.232062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.1.221.189:32985)
0810 22:33:22.232230 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:22.232234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:22.232355 (+ 121us) server_negotiation.cc:407] Connection header received
0810 22:33:22.232490 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:22.232493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:22.232542 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:22.232650 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:22.233138 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35554: BlockingRecv error: recv got EOF from 127.0.0.1:35554 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:33:22.538430 27100 catalog_manager.cc:1259] Loaded cluster ID: b93f33947c3e420eb342c74fe979acd5
I20260810 22:33:22.538520 27100 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad: loading cluster ID for follower catalog manager: success
I20260810 22:33:22.540118 27100 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad: acquiring CA information for follower catalog manager: success
I20260810 22:33:22.541226 27100 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f5d5871a240b4f2b8ba5965c2501c5ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:33:22.599661 27126 catalog_manager.cc:1259] Loaded cluster ID: b93f33947c3e420eb342c74fe979acd5
I20260810 22:33:22.599736 27126 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0: loading cluster ID for follower catalog manager: success
I20260810 22:33:22.601365 27126 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0: acquiring CA information for follower catalog manager: success
I20260810 22:33:22.602490 27126 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bffda888a43f4aaabdad8a41e9c7b5d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:33:22.751487 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:22.747399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.1.221.189:32985)
0810 22:33:22.747712 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:22.747716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:22.747733 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:33:22.747888 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:22.747892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:22.748035 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:22.748161 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:22.748448 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:22.748982 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:22.749918 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:22.750170 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:22.750285 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:22.750622 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:22.751069 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:22.751227 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:22.751261 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 22:33:22.751319 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":115,"threads_started":1}
I20260810 22:33:22.758459 27210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:22.754827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.1.221.188:35957)
0810 22:33:22.755115 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:22.755120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:22.755134 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:33:22.755222 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:22.755224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:22.755340 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:22.755440 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:22.755603 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:22.756127 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:22.756956 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:22.757186 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:22.757264 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:22.757551 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:22.758052 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:22.758155 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:22.758206 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 22:33:22.758255 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":101,"threads_started":1}
I20260810 22:33:22.765551 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:22.761510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.1.221.189:32985)
0810 22:33:22.761680 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:22.761687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:22.761890 (+ 203us) server_negotiation.cc:407] Connection header received
0810 22:33:22.761947 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:22.761950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:22.762005 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:22.762103 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:22.762400 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:22.763109 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:22.764499 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:22.764703 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:22.764806 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:22.765091 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:22.765174 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:22.765263 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:22.765356 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 22:33:22.765404 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:33:22.765630 27210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:22.761920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.1.221.188:35957)
0810 22:33:22.762041 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:22.762046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:22.762410 (+ 364us) server_negotiation.cc:407] Connection header received
0810 22:33:22.762560 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:22.762563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:22.762611 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:22.762683 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:22.762936 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:22.763498 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:22.764388 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:22.764718 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:22.764808 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:22.765162 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:22.765236 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:22.765301 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:22.765439 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 22:33:22.765486 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 22:33:22.766230 27216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:22.761183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.1.221.190:44259)
0810 22:33:22.761433 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:22.761437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:22.761453 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:33:22.761628 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:22.761633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:22.761804 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:22.762074 (+ 270us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:22.762294 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:22.763091 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:22.764479 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:22.764815 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:22.764909 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:22.765245 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:22.765569 (+ 324us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:22.765989 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:22.766026 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 22:33:22.766074 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
I20260810 22:33:22.768957 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 26825
I20260810 22:33:22.777942 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 26908
I20260810 22:33:22.786317 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 26987
2026-08-10T22:33:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3533 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:33:22 dist-test-slave-njxd krb5kdc[27221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:33:22 dist-test-slave-njxd krb5kdc[27221](info): set up 2 sockets
Aug 10 22:33:22 dist-test-slave-njxd krb5kdc[27221](info): commencing operation
krb5kdc: starting...
W20260810 22:33:23.894986 1910 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/test-user.keytab.
Aug 10 22:33:23 dist-test-slave-njxd krb5kdc[27221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:33:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:33:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.221.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
Entry for principal kudu/127.1.221.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab.
I20260810 22:33:24.024507 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40109
--webserver_interface=127.1.221.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40109,127.1.221.189:44209,127.1.221.188:45105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:33:24.137215 27235 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 22:33:24.137554 27235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:24.137652 27235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:24.137734 27235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:24.141726 27235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:24.141824 27235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:24.141862 27235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:24.141901 27235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:24.141935 27235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:24.147162 27235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40109,127.1.221.189:44209,127.1.221.188:45105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.190.keytab
--principal=kudu/127.1.221.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.190:40109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.221.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:24.148293 27235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:24.149379 27235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:24.157994 27240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:24.158067 27243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:24.158026 27235 server_base.cc:1034] running on GCE node
W20260810 22:33:24.157996 27241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:24.158871 27235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:24.159945 27235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:24.161142 27235 hybrid_clock.cc:648] HybridClock initialized: now 1786401204161121 us; error 44 us; skew 500 ppm
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:24.163998 27235 init.cc:377] Logged in from keytab as kudu/127.1.221.190@KRBTEST.COM (short username kudu)
I20260810 22:33:24.164934 27235 webserver.cc:460] Webserver started at http://127.1.221.190:37201/ using document root <none> and password file <none>
I20260810 22:33:24.165498 27235 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:24.165611 27235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:24.165853 27235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:24.167891 27235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/data/instance:
uuid: "f6827e5463a04b038b81bff1607b712c"
format_stamp: "Formatted at 2026-08-10 22:33:24 on dist-test-slave-njxd"
server_key: "c1b4b98bd892809132cf7bfce7edca3b"
server_key_iv: "75e3cf6da498c6965d4d877493761efe"
server_key_version: "encryptionkey@0"
I20260810 22:33:24.168488 27235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/wal/instance:
uuid: "f6827e5463a04b038b81bff1607b712c"
format_stamp: "Formatted at 2026-08-10 22:33:24 on dist-test-slave-njxd"
server_key: "c1b4b98bd892809132cf7bfce7edca3b"
server_key_iv: "75e3cf6da498c6965d4d877493761efe"
server_key_version: "encryptionkey@0"
I20260810 22:33:24.172423 27235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260810 22:33:24.175012 27250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:24.176234 27235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:33:24.176373 27235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/wal
uuid: "f6827e5463a04b038b81bff1607b712c"
format_stamp: "Formatted at 2026-08-10 22:33:24 on dist-test-slave-njxd"
server_key: "c1b4b98bd892809132cf7bfce7edca3b"
server_key_iv: "75e3cf6da498c6965d4d877493761efe"
server_key_version: "encryptionkey@0"
I20260810 22:33:24.176484 27235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:24.197240 27235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:24.200510 27235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:24.200718 27235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:24.207839 27235 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.190:40109
I20260810 22:33:24.208123 27308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.190:40109 every 8 connection(s)
I20260810 22:33:24.208797 27235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/data/info.pb
I20260810 22:33:24.211449 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 27235
I20260810 22:33:24.211588 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-0/wal/instance
I20260810 22:33:24.211915 1910 external_mini_cluster.cc:1428] Setting key eb9e93a1f2b8aabb18e551d6cdc7e011
I20260810 22:33:24.212132 27309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:24.216352 27309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.221.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.189@KRBTEST.COM" created.
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:33:24.231689 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.218711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40109 (local address 127.0.0.1:44694)
0810 22:33:24.219299 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.219335 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:24.220192 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:24.220897 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:24.220903 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:24.221356 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:24.221657 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.221678 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.222872 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.222875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.223593 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.223601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.224280 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.224284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.224393 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.224970 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:24.224986 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:24.226555 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:24.228626 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.228636 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.228646 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.228883 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.229285 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.229287 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.229288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.229399 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.230253 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:24.230261 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:24.230402 (+ 141us) client_negotiation.cc:769] Sending connection context
0810 22:33:24.230548 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 22:33:24.230830 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"spinlock_wait_cycles":17792,"thread_start_us":150,"threads_started":1}
I20260810 22:33:24.231705 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.218710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.1.221.190:40109)
0810 22:33:24.219142 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.219163 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.219444 (+ 281us) server_negotiation.cc:407] Connection header received
0810 22:33:24.220508 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.220513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.220711 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.220900 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.221811 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.222740 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.223720 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.224082 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.224293 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.226714 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.226721 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.226740 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.226760 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.228455 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.229010 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.229019 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.229024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.229178 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.229497 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.229503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.229508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.230111 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.230272 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.230686 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.230972 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":282,"thread_start_us":140,"threads_started":1}
I20260810 22:33:24.234165 27309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:24.234937 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.234676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:44209 (local address 127.0.0.1:52736)
0810 22:33:24.234800 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.234878 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:44209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 22:33:24.235189 27251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.189:44209: connect: Connection refused (error 111)
I20260810 22:33:24.236123 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.235885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:44209 (local address 127.0.0.1:52740)
0810 22:33:24.236027 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.236072 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:44209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 22:33:24.236315 27309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:44209: Network error: Client connection negotiation failed: client connection to 127.1.221.189:44209: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.189.keytab.
Entry for principal kudu/127.1.221.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.189.keytab.
I20260810 22:33:24.244340 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:44209
--webserver_interface=127.1.221.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.189.keytab
--principal=kudu/127.1.221.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40109,127.1.221.189:44209,127.1.221.188:45105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:33:24.285604 27309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } attempt: 1
I20260810 22:33:24.286569 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.286239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:44209 (local address 127.0.0.1:52746)
0810 22:33:24.286416 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.286482 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:44209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 22:33:24.287328 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.287115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:44209 (local address 127.0.0.1:52756)
0810 22:33:24.287226 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.287278 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:44209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 22:33:24.287451 27309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:44209: Network error: Client connection negotiation failed: client connection to 127.1.221.189:44209: connect: Connection refused (error 111)
I20260810 22:33:24.355609 27309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } attempt: 2
I20260810 22:33:24.356472 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.356162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:44209 (local address 127.0.0.1:52770)
0810 22:33:24.356333 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.356398 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:44209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260810 22:33:24.356441 27318 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 22:33:24.356817 27318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:24.356880 27318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:24.356952 27318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 22:33:24.357249 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.357054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:44209 (local address 127.0.0.1:52778)
0810 22:33:24.357140 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.357196 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.189:44209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260810 22:33:24.357400 27309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.189:44209: Network error: Client connection negotiation failed: client connection to 127.1.221.189:44209: connect: Connection refused (error 111)
W20260810 22:33:24.361090 27318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:24.361168 27318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:24.361210 27318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:24.361245 27318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:24.361281 27318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:24.366616 27318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40109,127.1.221.189:44209,127.1.221.188:45105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.189.keytab
--principal=kudu/127.1.221.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.189:44209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.221.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:24.367650 27318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:24.368698 27318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:24.377127 27329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:24.377068 27326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:24.377064 27325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:24.377429 27318 server_base.cc:1034] running on GCE node
I20260810 22:33:24.378046 27318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:24.379072 27318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:24.380280 27318 hybrid_clock.cc:648] HybridClock initialized: now 1786401204380259 us; error 57 us; skew 500 ppm
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:24.383083 27318 init.cc:377] Logged in from keytab as kudu/127.1.221.189@KRBTEST.COM (short username kudu)
I20260810 22:33:24.383898 27318 webserver.cc:460] Webserver started at http://127.1.221.189:39873/ using document root <none> and password file <none>
I20260810 22:33:24.384454 27318 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:24.384545 27318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:24.384778 27318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:24.386866 27318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/data/instance:
uuid: "4eca06f4ba064dd5bb68e62043688148"
format_stamp: "Formatted at 2026-08-10 22:33:24 on dist-test-slave-njxd"
server_key: "32b7dad688b39209a0642ea01542db68"
server_key_iv: "a15a6824e28fe0452e98dbd4caa423be"
server_key_version: "encryptionkey@0"
I20260810 22:33:24.387427 27318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/wal/instance:
uuid: "4eca06f4ba064dd5bb68e62043688148"
format_stamp: "Formatted at 2026-08-10 22:33:24 on dist-test-slave-njxd"
server_key: "32b7dad688b39209a0642ea01542db68"
server_key_iv: "a15a6824e28fe0452e98dbd4caa423be"
server_key_version: "encryptionkey@0"
I20260810 22:33:24.391065 27318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:33:24.393501 27337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:24.394793 27318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:33:24.394932 27318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/wal
uuid: "4eca06f4ba064dd5bb68e62043688148"
format_stamp: "Formatted at 2026-08-10 22:33:24 on dist-test-slave-njxd"
server_key: "32b7dad688b39209a0642ea01542db68"
server_key_iv: "a15a6824e28fe0452e98dbd4caa423be"
server_key_version: "encryptionkey@0"
I20260810 22:33:24.395040 27318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:24.414098 27318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:24.416841 27318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:24.417040 27318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:24.424075 27318 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.189:44209
I20260810 22:33:24.424382 27390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.189:44209 every 8 connection(s)
I20260810 22:33:24.425172 27318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/data/info.pb
I20260810 22:33:24.427934 27391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:24.431180 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 27318
I20260810 22:33:24.431320 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-1/wal/instance
I20260810 22:33:24.431618 1910 external_mini_cluster.cc:1428] Setting key 189df0fca299b8238a4e048a3f68f142
I20260810 22:33:24.431869 27391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.221.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.221.188@KRBTEST.COM" created.
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
I20260810 22:33:24.447708 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.433755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44710 (local address 127.1.221.190:40109)
0810 22:33:24.433958 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.433964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.434837 (+ 873us) server_negotiation.cc:407] Connection header received
0810 22:33:24.435779 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.435782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.435840 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.435951 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.437218 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.437851 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.439043 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.439331 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.439444 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.443847 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.443850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.443852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.443878 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.445446 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.446024 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.446027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.446029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.446115 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.446475 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.446478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.446480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.446682 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.446797 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.447384 (+ 587us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.447520 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 22:33:24.448546 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.434152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40109 (local address 127.0.0.1:44710)
0810 22:33:24.434649 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.434706 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:24.435571 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:24.436125 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:24.436132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:24.436583 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:24.437063 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.437077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.437983 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.437992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.438871 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.438880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.439501 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.439505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.439679 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.440347 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:24.440363 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:24.443659 (+ 3296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:24.445610 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.445616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.445633 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.445912 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.446215 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.446219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.446221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.446372 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.446796 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:24.446805 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:24.447097 (+ 292us) client_negotiation.cc:769] Sending connection context
0810 22:33:24.447309 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 22:33:24.447656 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":134,"threads_started":1}
I20260810 22:33:24.448558 27309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } attempt: 3
I20260810 22:33:24.450328 27391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.221.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.188.keytab.
Entry for principal kudu/127.1.221.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.188.keytab.
I20260810 22:33:24.460489 1910 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
/tmp/dist-test-taskYvFf7I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:45105
--webserver_interface=127.1.221.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.188.keytab
--principal=kudu/127.1.221.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.221.148:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.221.190:40109,127.1.221.189:44209,127.1.221.188:45105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:33:24.460996 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.450870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:44209 (local address 127.0.0.1:52794)
0810 22:33:24.450989 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.451006 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:24.451088 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:24.451529 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:24.451534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:24.451712 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:24.451822 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.451829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.452832 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.452836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.453389 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.453397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.453933 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.453937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.454033 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.454402 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:24.454421 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:24.456086 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:24.458657 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.458661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.458662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.459026 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.459441 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.459446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.459451 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.459560 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.460275 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:24.460281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:24.460396 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 22:33:24.460515 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 22:33:24.460688 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 22:33:24.461042 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.450988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.1.221.189:44209)
0810 22:33:24.451295 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.451302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.451315 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:33:24.451348 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.451350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.451387 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.451471 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.451955 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.452706 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.453536 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.453825 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.453934 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.456262 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.456265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.456267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.456314 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.458535 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.459175 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.459182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.459184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.459335 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.459708 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.459711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.459713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.460148 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.460291 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.460632 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.460826 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":252,"spinlock_wait_cycles":5376,"thread_start_us":118,"threads_started":1}
I20260810 22:33:24.461023 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.448971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:44209 (local address 127.0.0.1:52780)
0810 22:33:24.449155 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.449171 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:24.449250 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:24.450095 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:24.450098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:24.450283 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:24.450375 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.450380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.451729 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.451734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.452690 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.452699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.453466 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.453470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.453631 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.454224 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:24.454244 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:24.455660 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:24.458764 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.458767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.458769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.459027 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.459512 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.459516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.459517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.459614 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.460278 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:24.460281 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:24.460394 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 22:33:24.460462 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 22:33:24.460874 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260810 22:33:24.462090 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.449348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.1.221.189:44209)
0810 22:33:24.449746 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.449756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.449774 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:33:24.449811 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.449815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.449965 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.450114 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.450490 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.451604 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.452823 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.453309 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.453463 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.455830 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.455840 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.455857 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.455883 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.458654 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.459179 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.459183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.459184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.459341 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.459750 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.459755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.459759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.460148 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.461757 (+ 1609us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.461822 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.461914 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":282,"spinlock_wait_cycles":9088,"thread_start_us":169,"threads_started":1}
I20260810 22:33:24.463259 27309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:24.463340 27391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:24.464033 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.463799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49828)
0810 22:33:24.463914 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.463970 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 22:33:24.464159 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.463815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49834)
0810 22:33:24.463994 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.464075 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260810 22:33:24.464344 27339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
I20260810 22:33:24.464783 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.464492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49842)
0810 22:33:24.464648 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.464706 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260810 22:33:24.464952 27309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:45105: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
I20260810 22:33:24.465281 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.464995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49858)
0810 22:33:24.465178 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.465232 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260810 22:33:24.465430 27391 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:45105: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
I20260810 22:33:24.496111 27309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } attempt: 1
I20260810 22:33:24.497077 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.496673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49872)
0810 22:33:24.496878 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.496979 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 22:33:24.497929 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.497655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49878)
0810 22:33:24.497787 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.497850 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 22:33:24.498087 27309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:45105: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
I20260810 22:33:24.514808 27391 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } attempt: 1
I20260810 22:33:24.515600 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.515313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49884)
0810 22:33:24.515451 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.515530 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 22:33:24.516355 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.516138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49892)
0810 22:33:24.516250 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.516297 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 22:33:24.516587 27391 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:45105: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
I20260810 22:33:24.573264 27309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } attempt: 2
I20260810 22:33:24.574205 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.573881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49898)
0810 22:33:24.574059 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.574123 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 22:33:24.575143 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.574849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49906)
0810 22:33:24.575002 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.575064 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 22:33:24.575349 27309 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:45105: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
W20260810 22:33:24.579133 27398 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 22:33:24.579455 27398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:33:24.579500 27398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:33:24.579597 27398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:33:24.583943 27398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:33:24.584002 27398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:33:24.584052 27398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:33:24.584095 27398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:33:24.584134 27398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:33:24.584746 27391 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } attempt: 2
I20260810 22:33:24.585510 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.585245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49922)
0810 22:33:24.585384 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.585440 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 22:33:24.586215 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.586013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49934)
0810 22:33:24.586110 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.586152 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 22:33:24.586414 27391 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.221.188:45105: Network error: Client connection negotiation failed: client connection to 127.1.221.188:45105: connect: Connection refused (error 111)
I20260810 22:33:24.590858 27398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.221.148:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.221.190:40109,127.1.221.189:44209,127.1.221.188:45105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/krb5kdc/kudu_127.1.221.188.keytab
--principal=kudu/127.1.221.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.221.188:45105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.221.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 21:55:49 UTC on ccycloud.quasar-ktihrd.root.comops.site
build id 4579236
I20260810 22:33:24.591935 27398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:33:24.593003 27398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:33:24.601963 27408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:24.602057 27398 server_base.cc:1034] running on GCE node
W20260810 22:33:24.601970 27405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:33:24.602276 27404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:33:24.602875 27398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:33:24.604023 27398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:33:24.605232 27398 hybrid_clock.cc:648] HybridClock initialized: now 1786401204605196 us; error 53 us; skew 500 ppm
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:33:24.608155 27398 init.cc:377] Logged in from keytab as kudu/127.1.221.188@KRBTEST.COM (short username kudu)
I20260810 22:33:24.609087 27398 webserver.cc:460] Webserver started at http://127.1.221.188:36807/ using document root <none> and password file <none>
I20260810 22:33:24.609724 27398 fs_manager.cc:359] Metadata directory not provided
I20260810 22:33:24.609819 27398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:33:24.610070 27398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:33:24.612159 27398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/data/instance:
uuid: "5b6fd04a298b43618aa1a7cb4fa67e71"
format_stamp: "Formatted at 2026-08-10 22:33:24 on dist-test-slave-njxd"
server_key: "9f3352db95f8c8f5fcf49afcbe058b1a"
server_key_iv: "29e60335f998fe343359b3e8ba73b408"
server_key_version: "encryptionkey@0"
I20260810 22:33:24.612821 27398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/wal/instance:
uuid: "5b6fd04a298b43618aa1a7cb4fa67e71"
format_stamp: "Formatted at 2026-08-10 22:33:24 on dist-test-slave-njxd"
server_key: "9f3352db95f8c8f5fcf49afcbe058b1a"
server_key_iv: "29e60335f998fe343359b3e8ba73b408"
server_key_version: "encryptionkey@0"
I20260810 22:33:24.616729 27398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:33:24.619270 27416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:33:24.620528 27398 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:33:24.620684 27398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/data,/tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/wal
uuid: "5b6fd04a298b43618aa1a7cb4fa67e71"
format_stamp: "Formatted at 2026-08-10 22:33:24 on dist-test-slave-njxd"
server_key: "9f3352db95f8c8f5fcf49afcbe058b1a"
server_key_iv: "29e60335f998fe343359b3e8ba73b408"
server_key_version: "encryptionkey@0"
I20260810 22:33:24.620810 27398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:33:24.637153 27398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:33:24.640532 27398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:33:24.640751 27398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:33:24.648335 27398 rpc_server.cc:307] RPC server started. Bound to: 127.1.221.188:45105
I20260810 22:33:24.648636 27473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.221.188:45105 every 8 connection(s)
I20260810 22:33:24.649433 27398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/data/info.pb
I20260810 22:33:24.652328 27474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:33:24.656139 27474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:24.659466 1910 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu as pid 27398
I20260810 22:33:24.659614 1910 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYvFf7I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786400913154531-1910-0/minicluster-data/master-2/wal/instance
I20260810 22:33:24.659941 1910 external_mini_cluster.cc:1428] Setting key b51978f1bfd2e2dfd6deb0d6942fa130
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260810 22:33:24.670800 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.658140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.1.221.190:40109)
0810 22:33:24.658285 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.658289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.659151 (+ 862us) server_negotiation.cc:407] Connection header received
0810 22:33:24.660456 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.660460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.660523 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.660612 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.661973 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.662499 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.663801 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.664036 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.664212 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.666965 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.666968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.666970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.666997 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.668613 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.669191 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.669195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.669197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.669306 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.669675 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.669679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.669682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.669883 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.669996 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.670492 (+ 496us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.670620 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:33:24.671605 27476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.658527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.190:40109 (local address 127.0.0.1:44712)
0810 22:33:24.658987 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.659034 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:24.660200 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:24.660878 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:24.660888 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:24.661279 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:24.661818 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.661834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.662615 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.662620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.663658 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.663668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.664234 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.664239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.664474 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.665109 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:24.665134 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:24.666800 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:24.668722 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.668732 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.668742 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.669081 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.669407 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.669410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.669412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.669553 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.669995 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:24.670003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:24.670260 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 22:33:24.670430 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 22:33:24.670753 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":235,"thread_start_us":122,"threads_started":1}
I20260810 22:33:24.671605 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.660690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.1.221.188:45105)
0810 22:33:24.661095 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.661107 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.661131 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:33:24.661228 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.661233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.661404 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.661570 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.662387 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.663280 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.664185 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.664587 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.664713 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.667363 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.667371 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.667388 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.667415 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.669144 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.669650 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.669655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.669657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.669731 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.670012 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.670016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.670018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.670276 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.670490 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.670678 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.671046 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":301,"thread_start_us":155,"threads_started":1}
I20260810 22:33:24.672827 27474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:24.674486 27309 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } attempt: 3
I20260810 22:33:24.677618 27391 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } attempt: 3
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.190@KRBTEST.COM
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.190@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260810 22:33:24.684017 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.673501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52796 (local address 127.1.221.189:44209)
0810 22:33:24.673671 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.673675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.673689 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:33:24.673764 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.673768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.673824 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.673919 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.674398 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.675077 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.676539 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.676887 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.676983 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.679602 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.679606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.679609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.679657 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.681378 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.681914 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.681920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.681925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.682014 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.683059 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.683062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.683064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.683257 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.683367 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.683686 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.683825 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.189@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
I20260810 22:33:24.684780 27476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.673313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.189:44209 (local address 127.0.0.1:52796)
0810 22:33:24.673477 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.673490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:24.673562 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:24.673957 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:24.673961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:24.674155 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:24.674276 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.674282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.675252 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.675256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.676020 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.676029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.677095 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.677098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.677208 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.677715 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:24.677729 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:24.679450 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:24.681500 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.681505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.681509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.681804 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.682112 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.682117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.682119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.682220 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.683358 (+ 1138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:24.683362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:24.683454 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 22:33:24.684410 (+ 956us) client_negotiation.cc:240] Negotiation successful
0810 22:33:24.684580 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260810 22:33:24.686142 27474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:33:24.686620 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.674069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.1.221.190:40109)
0810 22:33:24.674167 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.674170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.674183 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:33:24.674257 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.674259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.674290 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.674378 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.675111 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.675835 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.676648 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.676871 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.676981 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.680491 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.680497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.680502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.680530 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.682805 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.684303 (+ 1498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.684307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.684309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.684399 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.684736 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.684739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.684741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.684979 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.685201 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.686096 (+ 895us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.686447 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 22:33:24.687567 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.675469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49946)
0810 22:33:24.675578 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.675593 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:24.675663 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:24.676152 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:24.676155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:24.676323 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:24.676392 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.676398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.677338 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.677342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.678189 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.678199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.679943 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.679948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.680046 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.680843 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:24.680857 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:24.682947 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:24.685745 (+ 2798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.685748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.685750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.686017 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.686345 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.686348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.686350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.686444 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.687092 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:24.687095 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:24.687192 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 22:33:24.687301 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 22:33:24.687424 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 22:33:24.687578 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.675511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.1.221.188:45105)
0810 22:33:24.675973 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.675976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.675985 (+ 9us) server_negotiation.cc:407] Connection header received
0810 22:33:24.676010 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.676012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.676042 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.676137 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.676508 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.677195 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.679360 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.679803 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.680298 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.683430 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.683434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.683436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.683454 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.685452 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.686141 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.686145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.686147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.686228 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.686540 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.686546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.686549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.686783 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.686872 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.687355 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.687453 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403}
I20260810 22:33:24.690613 27394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.678024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49950)
0810 22:33:24.678134 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.678148 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:24.678249 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:24.678685 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:24.678688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:24.678867 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:24.678968 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.678974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.681229 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.681235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.682111 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.682123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.682929 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.682933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.683037 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.683468 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:24.683481 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:24.685201 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:24.689017 (+ 3816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.689020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.689022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.689283 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.689661 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.689664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.689666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.689750 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.690169 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:24.690171 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:24.690254 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 22:33:24.690330 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 22:33:24.690451 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"spinlock_wait_cycles":6400}
I20260810 22:33:24.691314 27480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.678089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.1.221.188:45105)
0810 22:33:24.678443 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.678453 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.678469 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:33:24.678504 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.678507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.678549 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.678656 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.679087 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.681058 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.682322 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.682798 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.682909 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.685347 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.685354 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.685359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.685388 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.688903 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.689412 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.689415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.689417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.689507 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.689865 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.689867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.689869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.690056 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.690910 (+ 854us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.690985 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.691103 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
I20260810 22:33:24.692219 27309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c: Bootstrap starting.
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401204, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.221.188@KRBTEST.COM for kudu/127.1.221.188@KRBTEST.COM
Aug 10 22:33:24 dist-test-slave-njxd krb5kdc[27221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786401203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.221.189@KRBTEST.COM
I20260810 22:33:24.695122 27391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148: Bootstrap starting.
I20260810 22:33:24.696240 27476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.686582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.221.188:45105 (local address 127.0.0.1:49964)
0810 22:33:24.686954 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 22:33:24.686967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:33:24.687022 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:33:24.688032 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:33:24.688036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:33:24.688194 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:33:24.688269 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.688274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.689181 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.689184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.689981 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:33:24.689989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.690589 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.690592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:33:24.690898 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.691339 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:33:24.691539 (+ 200us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:33:24.692802 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:33:24.694245 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.694251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.694252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.694466 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.695313 (+ 847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:33:24.695316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:33:24.695318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:33:24.695409 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:33:24.695839 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:33:24.695841 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:33:24.695907 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 22:33:24.695993 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 22:33:24.696095 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":5}
I20260810 22:33:24.696431 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.686823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.1.221.188:45105)
0810 22:33:24.687796 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.687799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.687810 (+ 11us) server_negotiation.cc:407] Connection header received
0810 22:33:24.687840 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.687842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.687875 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.687943 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.688375 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.689061 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.690083 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.690415 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.690504 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.692915 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.692918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.692920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.692940 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.694152 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.694847 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.694850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.694853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.694944 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.695503 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.695506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.695508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.695697 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.695796 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.696071 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.696205 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":93,"threads_started":1}
I20260810 22:33:24.697908 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.687694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52798 (local address 127.1.221.189:44209)
0810 22:33:24.687787 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.687790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.687873 (+ 83us) server_negotiation.cc:407] Connection header received
0810 22:33:24.688091 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.688094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.688136 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.688220 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.688925 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.689570 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.690780 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.691082 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.691168 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.694349 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.694352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.694355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.694378 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.695701 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.696179 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.696182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.696184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.696265 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.696917 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.696920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.696921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.697118 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.697279 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.697659 (+ 380us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.697775 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 22:33:24.699045 27474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71: Bootstrap starting.
I20260810 22:33:24.700251 27309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:24.700831 27474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:24.701089 1910 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:33:24.701203 27309 log.cc:824] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:24.701653 27474 log.cc:824] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:24.701810 27391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148: Neither blocks nor log segments found. Creating new log.
I20260810 22:33:24.702808 27391 log.cc:824] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148: Log is configured to *not* fsync() on all Append() calls
I20260810 22:33:24.703550 27474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71: No bootstrap required, opened a new log
I20260810 22:33:24.703550 27309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c: No bootstrap required, opened a new log
I20260810 22:33:24.702335 27281 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f6827e5463a04b038b81bff1607b712c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44722
I20260810 22:33:24.704896 27391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148: No bootstrap required, opened a new log
I20260810 22:33:24.706494 27309 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } }
I20260810 22:33:24.706740 27309 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6827e5463a04b038b81bff1607b712c, State: Initialized, Role: FOLLOWER
I20260810 22:33:24.707050 27309 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } }
I20260810 22:33:24.707425 27474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } }
I20260810 22:33:24.707636 27474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6fd04a298b43618aa1a7cb4fa67e71, State: Initialized, Role: FOLLOWER
I20260810 22:33:24.707588 27485 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } } }
I20260810 22:33:24.707916 27474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } }
I20260810 22:33:24.707969 27485 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:33:24.708316 27309 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:24.708369 27281 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 22:33:24.708472 27281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:24.708755 27391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } }
I20260810 22:33:24.708946 27391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eca06f4ba064dd5bb68e62043688148, State: Initialized, Role: FOLLOWER
I20260810 22:33:24.709175 27474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:24.709229 27391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } }
I20260810 22:33:24.709800 27281 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } }
I20260810 22:33:24.710422 27391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:33:24.711519 27363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6827e5463a04b038b81bff1607b712c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4eca06f4ba064dd5bb68e62043688148"
I20260810 22:33:24.711751 27363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:24.713235 27363 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6827e5463a04b038b81bff1607b712c in term 1.
I20260810 22:33:24.711517 27446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6827e5463a04b038b81bff1607b712c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71"
I20260810 22:33:24.714260 27251 leader_election.cc:302] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eca06f4ba064dd5bb68e62043688148, f6827e5463a04b038b81bff1607b712c; no voters:
I20260810 22:33:24.714447 27446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:33:24.714865 27485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:33:24.715135 27485 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [term 1 LEADER]: Becoming Leader. State: Replica: f6827e5463a04b038b81bff1607b712c, State: Running, Role: LEADER
W20260810 22:33:24.715301 27510 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:33:24.715380 27510 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:33:24.715574 27485 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } }
I20260810 22:33:24.715865 27446 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6827e5463a04b038b81bff1607b712c in term 1.
I20260810 22:33:24.715978 27486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } } }
I20260810 22:33:24.716104 27486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:33:24.713479 27235 master_runner.cc:186] Error getting master registration for 127.1.221.189:44209: OK, instance_id { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" instance_seqno: 1786401204422169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 22:33:24.717412 27235 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 22:33:24.717609 27485 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6827e5463a04b038b81bff1607b712c. Latest consensus state: current_term: 1 leader_uuid: "f6827e5463a04b038b81bff1607b712c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } } }
I20260810 22:33:24.717751 27485 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:24.710711 27281 leader_election.cc:288] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [CANDIDATE]: Term 1 election: Requested vote from peers 4eca06f4ba064dd5bb68e62043688148 (127.1.221.189:44209), 5b6fd04a298b43618aa1a7cb4fa67e71 (127.1.221.188:45105)
W20260810 22:33:24.720708 27503 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:33:24.720826 27503 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:33:24.720950 27526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:33:24.721856 27526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 22:33:24.724320 27522 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:33:24.724395 27522 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:33:24.724789 27492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } } }
I20260810 22:33:24.724936 27492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:33:24.732957 27363 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148 [term 1 FOLLOWER]: Refusing update from remote peer f6827e5463a04b038b81bff1607b712c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:24.733016 27446 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71 [term 1 FOLLOWER]: Refusing update from remote peer f6827e5463a04b038b81bff1607b712c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:33:24.733562 27485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:24.734016 27485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:33:24.737453 27486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6827e5463a04b038b81bff1607b712c. Latest consensus state: current_term: 1 leader_uuid: "f6827e5463a04b038b81bff1607b712c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } } }
I20260810 22:33:24.737605 27486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:33:24.737965 27492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6827e5463a04b038b81bff1607b712c. Latest consensus state: current_term: 1 leader_uuid: "f6827e5463a04b038b81bff1607b712c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } } }
I20260810 22:33:24.738087 27492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:33:24.743108 27486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6827e5463a04b038b81bff1607b712c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } } }
I20260810 22:33:24.743227 27486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:33:24.743918 27485 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6827e5463a04b038b81bff1607b712c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } } }
I20260810 22:33:24.744028 27485 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:24.744151 27533 mvcc.cc:205] Tried to move back new op lower bound from 7317099334579466240 to 7317099334519476224. Current Snapshot: MvccSnapshot[applied={T|T < 7317099334579466240}]
I20260810 22:33:24.747625 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.730136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.1.221.190:40109)
0810 22:33:24.730278 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.730286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.738054 (+ 7768us) server_negotiation.cc:407] Connection header received
0810 22:33:24.738340 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.738343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.738393 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.738443 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.739466 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.740265 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.741465 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.741816 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.741892 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.743886 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.743890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.743893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.743919 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.745761 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.746264 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.746268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.746271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.746355 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.746641 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.746644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.746646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.746862 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.746963 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.747335 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.747462 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:33:24.748965 27536 mvcc.cc:205] Tried to move back new op lower bound from 7317099334579466240 to 7317099334519476224. Current Snapshot: MvccSnapshot[applied={T|T < 7317099334579466240}]
I20260810 22:33:24.753388 27523 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6827e5463a04b038b81bff1607b712c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } } }
I20260810 22:33:24.753549 27523 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c [sys.catalog]: This master's current role is: LEADER
I20260810 22:33:24.754827 27492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6827e5463a04b038b81bff1607b712c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6827e5463a04b038b81bff1607b712c" member_type: VOTER last_known_addr { host: "127.1.221.190" port: 40109 } } peers { permanent_uuid: "4eca06f4ba064dd5bb68e62043688148" member_type: VOTER last_known_addr { host: "127.1.221.189" port: 44209 } } peers { permanent_uuid: "5b6fd04a298b43618aa1a7cb4fa67e71" member_type: VOTER last_known_addr { host: "127.1.221.188" port: 45105 } } }
I20260810 22:33:24.754971 27492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:33:24.755278 27537 mvcc.cc:205] Tried to move back new op lower bound from 7317099334579466240 to 7317099334519476224. Current Snapshot: MvccSnapshot[applied={T|T < 7317099334579466240}]
I20260810 22:33:24.757742 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.731555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.1.221.189:44209)
0810 22:33:24.731664 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.731669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.748808 (+ 17139us) server_negotiation.cc:407] Connection header received
0810 22:33:24.748892 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.748895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.748952 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.749035 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.750036 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.750733 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.751792 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.752111 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.752201 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.754056 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.754060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.754063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.754088 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.755879 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.756410 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.756414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.756416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.756499 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.756780 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.756784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.756786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.756986 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.757091 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.757425 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.757541 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 22:33:24.760881 27526 catalog_manager.cc:1347] Generated new cluster ID: c418c0bdd04a4de1bf479ce3c1cc4463
I20260810 22:33:24.760946 27526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:33:24.761945 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:24.730886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.1.221.188:45105)
0810 22:33:24.730991 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.730995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.740882 (+ 9887us) server_negotiation.cc:407] Connection header received
0810 22:33:24.744638 (+ 3756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.744642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.744700 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.744749 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.745606 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.746380 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.747545 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:24.747893 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:24.747965 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:24.758008 (+ 10043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:33:24.758012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:33:24.758015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:33:24.758044 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:33:24.759998 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.760495 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.760499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.760501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.760598 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:33:24.760896 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:33:24.760899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:33:24.760901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:33:24.761114 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:33:24.761224 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:24.761664 (+ 440us) server_negotiation.cc:299] Negotiation successful
0810 22:33:24.761785 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 22:33:24.794862 27526 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:33:24.796196 27526 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:33:24.818723 27526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6827e5463a04b038b81bff1607b712c: Generated new TSK 0
I20260810 22:33:24.819669 27526 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:33:24.856133 27312 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:24.852736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44734 (local address 127.1.221.190:40109)
0810 22:33:24.852878 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.852883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.854951 (+ 2068us) server_negotiation.cc:407] Connection header received
0810 22:33:24.855170 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.855174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.855230 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.855290 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.855991 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44734: BlockingRecv error: recv got EOF from 127.0.0.1:44734 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 22:33:24.857697 27479 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:24.853507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.1.221.188:45105)
0810 22:33:24.853651 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.853656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.856668 (+ 3012us) server_negotiation.cc:407] Connection header received
0810 22:33:24.856865 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.856868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.856921 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.856972 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.857497 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49978: BlockingRecv error: recv got EOF from 127.0.0.1:49978 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 22:33:24.859272 27396 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:24.854262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52814 (local address 127.1.221.189:44209)
0810 22:33:24.854386 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.854390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.858110 (+ 3720us) server_negotiation.cc:407] Connection header received
0810 22:33:24.858458 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.858463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.858524 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.858577 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.859144 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52814: BlockingRecv error: recv got EOF from 127.0.0.1:52814 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 22:33:24.864182 27396 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:24.862595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.1.221.189:44209)
0810 22:33:24.862726 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.862730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.863168 (+ 438us) server_negotiation.cc:407] Connection header received
0810 22:33:24.863395 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.863399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.863483 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.863538 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.864077 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52828: BlockingRecv error: recv got EOF from 127.0.0.1:52828 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 22:33:24.869490 27396 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:24.867980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.1.221.189:44209)
0810 22:33:24.868079 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.868083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.868546 (+ 463us) server_negotiation.cc:407] Connection header received
0810 22:33:24.868763 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.868769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.868833 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.868890 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.869391 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52838: BlockingRecv error: recv got EOF from 127.0.0.1:52838 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 22:33:24.875433 27396 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:24.873890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.1.221.189:44209)
0810 22:33:24.873988 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.873993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.874457 (+ 464us) server_negotiation.cc:407] Connection header received
0810 22:33:24.874693 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.874697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.874750 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.874805 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.875330 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52840: BlockingRecv error: recv got EOF from 127.0.0.1:52840 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 22:33:24.883767 27396 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:24.881993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.1.221.189:44209)
0810 22:33:24.882147 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.882152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.882716 (+ 564us) server_negotiation.cc:407] Connection header received
0810 22:33:24.882943 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.882947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.883004 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.883062 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.883638 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52856: BlockingRecv error: recv got EOF from 127.0.0.1:52856 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260810 22:33:24.896479 27396 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:24.894835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.1.221.189:44209)
0810 22:33:24.894944 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.894948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.895522 (+ 574us) server_negotiation.cc:407] Connection header received
0810 22:33:24.895728 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.895731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.895782 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.895843 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.896347 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52864: BlockingRecv error: recv got EOF from 127.0.0.1:52864 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 22:33:24.916759 27396 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:24.915033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.1.221.189:44209)
0810 22:33:24.915157 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.915162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.915712 (+ 550us) server_negotiation.cc:407] Connection header received
0810 22:33:24.915938 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.915942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.915995 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.916047 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.916622 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52880: BlockingRecv error: recv got EOF from 127.0.0.1:52880 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 22:33:24.952744 27396 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:24.951509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52886 (local address 127.1.221.189:44209)
0810 22:33:24.951662 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:24.951667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:24.951836 (+ 169us) server_negotiation.cc:407] Connection header received
0810 22:33:24.951953 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:24.951957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:24.952011 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:24.952112 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:24.952634 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52886: BlockingRecv error: recv got EOF from 127.0.0.1:52886 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 22:33:25.020463 27396 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:25.019388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.1.221.189:44209)
0810 22:33:25.019524 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:25.019528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:25.019687 (+ 159us) server_negotiation.cc:407] Connection header received
0810 22:33:25.019804 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:25.019806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:25.019857 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:25.019941 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:25.020361 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52890: BlockingRecv error: recv got EOF from 127.0.0.1:52890 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 22:33:25.151944 27396 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:25.150768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.1.221.189:44209)
0810 22:33:25.150929 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:25.150936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:25.151063 (+ 127us) server_negotiation.cc:407] Connection header received
0810 22:33:25.151202 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:25.151204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:25.151255 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:25.151329 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:25.151808 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52902: BlockingRecv error: recv got EOF from 127.0.0.1:52902 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 22:33:25.412001 27396 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:33:25.410616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.1.221.189:44209)
0810 22:33:25.410798 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:25.410804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:25.410933 (+ 129us) server_negotiation.cc:407] Connection header received
0810 22:33:25.411162 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:25.411165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:25.411234 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:25.411338 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:33:25.411884 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52906: BlockingRecv error: recv got EOF from 127.0.0.1:52906 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:33:25.716722 27510 catalog_manager.cc:1259] Loaded cluster ID: c418c0bdd04a4de1bf479ce3c1cc4463
I20260810 22:33:25.716840 27510 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71: loading cluster ID for follower catalog manager: success
I20260810 22:33:25.718619 27510 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71: acquiring CA information for follower catalog manager: success
I20260810 22:33:25.719832 27510 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5b6fd04a298b43618aa1a7cb4fa67e71: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:33:25.725492 27522 catalog_manager.cc:1259] Loaded cluster ID: c418c0bdd04a4de1bf479ce3c1cc4463
I20260810 22:33:25.725554 27522 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148: loading cluster ID for follower catalog manager: success
I20260810 22:33:25.727131 27522 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148: acquiring CA information for follower catalog manager: success
I20260810 22:33:25.728209 27522 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4eca06f4ba064dd5bb68e62043688148: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:33:25.930065 27620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:25.926385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.1.221.189:44209)
0810 22:33:25.926630 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:25.926633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:25.926654 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:33:25.926812 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:25.926815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:25.926945 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:25.927056 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:25.927305 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:25.927854 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:25.928685 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:25.928911 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:25.928990 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:25.929274 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:25.929734 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:25.929826 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:25.929890 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 22:33:25.929939 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":88,"threads_started":1}
I20260810 22:33:25.937275 27626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:25.933206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.1.221.188:45105)
0810 22:33:25.933463 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:25.933468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:25.933484 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:33:25.933669 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:25.933672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:25.933798 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:25.933929 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:25.934154 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:25.934876 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:25.935766 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:25.935999 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:25.936108 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:25.936432 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:25.936861 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:25.937006 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:25.937051 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 22:33:25.937108 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260810 22:33:25.945114 27626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:25.940826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.1.221.188:45105)
0810 22:33:25.940943 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:25.940949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:25.941215 (+ 266us) server_negotiation.cc:407] Connection header received
0810 22:33:25.941393 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:25.941396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:25.941449 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:25.941495 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:25.941904 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:25.942621 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:25.943723 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:25.943936 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:25.944053 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:25.944522 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:25.944619 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:25.944672 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:25.944883 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 22:33:25.944943 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 22:33:25.945794 27620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:25.940424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52932 (local address 127.1.221.189:44209)
0810 22:33:25.940557 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:25.940561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:25.940891 (+ 330us) server_negotiation.cc:407] Connection header received
0810 22:33:25.941343 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:25.941348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:25.941416 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:25.941490 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:25.942739 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:25.943236 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:25.944461 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:25.944791 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:25.944868 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:25.945244 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:25.945331 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:25.945380 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:25.945567 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 22:33:25.945654 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:33:25.945971 27634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:33:25.940391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.1.221.190:40109)
0810 22:33:25.940680 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0810 22:33:25.940685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:33:25.940916 (+ 231us) server_negotiation.cc:407] Connection header received
0810 22:33:25.941056 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:33:25.941059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:33:25.941234 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:33:25.941351 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:33:25.941631 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:25.942388 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:25.943567 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:33:25.943903 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:33:25.944003 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:33:25.944778 (+ 775us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:33:25.945268 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:33:25.945670 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0810 22:33:25.945709 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 22:33:25.945778 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260810 22:33:25.949115 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 27235
I20260810 22:33:25.961396 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 27318
I20260810 22:33:25.969210 1910 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYvFf7I/build/debug/bin/kudu with pid 27398
2026-08-10T22:33:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3180 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6714 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (292779 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST